Forem: Foteini Savvidou

Building an educational game with AI tools and Azure Static Web Apps (Part 2)

Foteini Savvidou — Wed, 08 Jan 2025 20:30:16 +0000

In Part 1, we explored how Ren'Py, a visual novel engine built on Python, and AI tools like GitHub Copilot, Azure OpenAI Service, and Microsoft Designer can be used to create a prototype for an educational game. In this post, I will share how I:

Created a GitHub Actions workflow to automate the build and deployment process for the game.
Used Azure Static Web Apps preview environments to review changes in the game before deploying them to production.

Curious to see the result? You can play the game online and find the source code at my GitHub repository.

Creating a CI/CD workflow

Manually building and deploying the game with each update can quickly become tedious. Fortunately, Ren'Py includes a CLI tool for automation, and Azure Static Web Apps integrates seamlessly with GitHub Actions, making it possible to automate the entire process.

My goal was to create a simple workflow that would meet the following requirements:

Automatically build the web version of the game and push it to a separate branch (e.g., the gh-pages branch) for builds triggered by the main branch.
Deploy the game to the production environment after a successful build from the main branch.
Provide the option to manually trigger builds and deploy to a preview or production environment from any branch.

Below, I outline the approach I followed. While this is not necessarily the best practice, it serves as documentation for the process I followed, as this was one of my first experiences creating a complete CI/CD workflow. The diagram below provides a high-level overview of the workflow:

Workflow diagram for automating the build and deployment of the game to Azure Static Web Apps using GitHub Actions.

The source code for the game is stored in the main branch of my GitHub repository.
When a new commit is pushed to the main branch, a GitHub Actions workflow is triggered. This workflow builds the web version of the game and deploys it to Azure Static Web Apps.
Additionally, the workflow can be triggered manually to build and deploy the game to a preview environment, allowing testing before merging changes into the main branch.

What is Azure Static Web Apps?

Azure Static Web Apps (SWA) is a cloud service designed to simplify the deployment and hosting of modern web applications. A static website consists of pre-rendered HTML, CSS, JavaScript, and media files that don't require server-side rendering. Azure SWA automates the process of building and deploying the web app by integrating directly with GitHub Actions. Whenever changes are pushed to a monitored branch or a pull request is opened, a new version of the website can be built and deployed.

Azure SWA offers additional, powerful features like Azure Functions integration for APIs, database integration, and support for custom domains that go beyond the scope of this project. You can explore the available features in the Azure SWA documentation.

For this project, the Ren'Py CLI generates a web version of the game, which is then deployed to Azure SWA using the static-web-apps-deploy GitHub Action. Below is an example of the default parameters for this action:

- name: Deploy to Azure Static Web Apps
  uses: Azure/static-web-apps-deploy@v1
  with:
    azure_static_web_apps_api_token: ${{ secrets.AZURE_STATIC_WEB_APPS_API_TOKEN }}
    repo_token: ${{ secrets.GITHUB_TOKEN }}
    action: "upload"
    app_location: "/"
    api_location: ""
    output_location: "/"

Setting up a preview environment in Azure SWA

Preview environments are an essential feature of Azure Static Web Apps (SWA) that enable you to review changes to your application before pushing them to production. Azure SWA supports three types of preview environment deployments:

Pull requests: Each pull request deploys a preview version of your app to a temporary URL. Once the pull request is merged or closed, the temporary environment is removed. If you use the default GitHub Actions template provided by Azure SWA, this process is configured by default.
Branch: Changes made to branches other than the production branch can deploy a preview version of the app to a stable URL that includes the branch name. To enable branch preview environments, you can set the production_branch input parameter in the static-web-apps-deploy GitHub Action.
Named environment: Named environments allow you to create stable preview deployments with a custom name. These environments are ideal for staging or manual testing. To enable named environments, you need to set the deployment_environment input parameter in the static-web-apps-deploy GitHub Action to the environment's name.

For this project, I used the named environment feature as it is suitable for manual, controlled deployments. To implement this, I added the deployment_environment input parameter to the deployment task. This parameter is set to the preview environment's name or left empty for deploying to production.

Bringing it all together

The final GitHub Actions workflow that I created automates the game's build and deployment processes through the following jobs:

Build the game using Ren'Py CLI: The web version of the game is generated using the Ren'Py CLI. The generated files are pushed to the gh-pages branch. If the gh-pages branch does not exist, it is created.
Deploy the game to Azure Static Web Apps: The gh-pages branch is deployed to Azure Static Web Apps using the Azure Static Web Apps GitHub Action described in the previous section.

The diagram below provides a high-level overview of the steps executed in the workflow.

High-level overview of the steps for building the web version of the game with the Ren'Py CLI and deploying it to Azure Static Web Apps.

There are two possible triggers for the workflow:

Automatic trigger on push to main: The workflow is triggered automatically when a new commit is pushed to the main branch. The workflow builds the game and deploys it to the production environment.
Manual trigger: The workflow can be triggered manually to build and deploy the game to either a preview or production environment from any branch. The input parameters required include the source branch (containing Ren'Py game files), target branch (for generated files), and environment name.

Summary

In this post, I described the steps I took to create a GitHub Actions workflow for automating the build and deployment of a Ren'Py game to Azure Static Web Apps. I also showed how preview environments can be used to test changes before they are deployed to production.

If you're interested in learning more about Azure Static Web Apps or GitHub Actions, check out the following courses on Microsoft Learn:

Building an educational game with AI tools and Azure Static Web Apps (Part 1)

Foteini Savvidou — Wed, 08 Jan 2025 20:23:04 +0000

Have you ever wondered how games can transform the way we learn? For me, the idea of blending creativity, technology, and a touch of fun has always been fascinating. Recently, I had the chance to explore this by building an educational visual novel game. Although I had no prior experience in game development, I was eager to try something new, so I took my first steps with Ren'Py, a visual novel engine built on Python. Using AI tools to speed up development and Azure Static Web Apps for deployment, I created a prototype for an interactive game that teaches you how to use GitHub Copilot to assist with coding tasks. Since it was December, I added a festive touch to the game to make the experience more fun.

In this post, I'll share how I:

Built the game using the Ren'Py framework.
Used AI tools like GitHub Copilot and Azure OpenAI Service to speed up development and generate the game's visual assets.
Automated the build and deployment process using GitHub Actions and Azure Static Web Apps.

Curious to see the result? You can play the game online and find the source code at my GitHub repository.

How it all started

There are several educational games about cloud technologies, covering everything from beginner to advanced users. For example, Microsoft offers Microsoft Technical Quest, a card-based game where you build a reference architecture using Azure services. Other cloud providers, like AWS, also have game-based training to help you learn how to build cloud solutions in a fun, gamified way.

A few years ago, I played a game called Azure Space Mystery, created by Microsoft Cloud Advocates for the International Day for Women and Girls in Science. It was a text-based game with rich graphics that included questions to move to the next level. The idea was to bring Microsoft Learn content closer to developers in a fun way.

I wanted to try building something similar, even though I had no experience in game development. My idea was to create a text-based game that teaches a technical concept, provides short quizzes, and rewards players with achievements as they progress. Since GitHub Copilot has been getting so much attention lately, I decided to make it the main focus of the game. Because I started working on the game in December, I also added a festive theme.

The result was Christmas Copilot Quest, a game where players learn how to use GitHub Copilot in Visual Studio Code to help with coding tasks. Guided by GingerBot, Santa's friendly Copilot-powered assistant, players are introduced to the features of GitHub Copilot in an interactive way.

Screenshots of the game showcasing the main menu, an example of dialogue, and the learning resources page.

Building a text-based game

The tech stack

There are many great tools for building text-based games, but I had a few key requirements in mind:

Support for writing non-linear stories with quizzes or choices that affect the flow of the game.
The ability to customize the game's user interface.
Flexibility to write custom components.
Support for deploying the game as a web app.

Since Python is my go-to language, I was naturally drawn to Ren'Py, a visual novel engine built on Python. It met all my requirements, offering an easy-to-use scripting language for writing the story, defining quizzes, and customizing the UI. It also provides the flexibility to extend its built-in functionality using Python and export the game to desktop, mobile, and web platforms. Additionally, the availability of the Ren'Py Command Line Interface (CLI) allowed me to automate the process of building and deploying the game.

The game structure

The game is built around three core components:

Script: This includes the game's story and quizzes. The narrative can take the form of monologues or dialogues between characters. The story is organized into sections identified by labels.

Screenshots of the game showcasing an example of dialogue and a quiz for the player to answer.

Graphical User Interface (GUI): This covers the screens and menus displayed throughout the game as well as the visual elements used. Ren'Py scripting language allows for both the customization of built-in screens (e.g., buttons and menus) and the creation of new ones, such as:

A notification screen to display information to the player when an achievement is unlocked.
A menu to show earned achievements and learning resources.

Custom game screens: character selection (left), achievement unlocked notification (center), and earned achievements screen (right).

Custom Python code: This provides additional functionality tailored to the game's needs. The latest version of Ren'Py supports scripting custom components using Python 3.9. Custom components in the game include:

An achievement system, allowing players to earn rewards for completing specific tasks, with achievements being stored persistently.
Character definitions and utility functions enabling players to select their preferred character and name.
GUI utilities, such as image transformations (e.g., eye blinks) and custom fonts and text styles written in Python.

To keep the game maintainable and extensible, these components were kept separate. The script defines the story, while the logic is organized into Python modules, which are called stores in Ren'Py. For example, a Python function that determines the player's name based on their input can be called directly from the script as follows:

label introduction:
    felix "Ah, you must be the new coder Santa called for! What's your name?"

    $ player_input = renpy.input(
        _("(Type your name and press Enter, or press Enter to use the default name, [character_name].)")
    )
    $ player_name = character_utils.determine_player_name(player_input)

    player "I'm [player_name]."

Integrating AI tools in development

Using GitHub Copilot for coding assistance

I found GitHub Copilot very helpful for navigating and understanding Ren'Py, a framework I had never worked with before. Although Ren'Py is not a widely used framework, and likely has limited data available for training, GitHub Copilot correctly answered most of my questions and helped me understand how various Ren'Py components work. One example was its suggestion for implementing a character selection screen with image buttons.

GitHub Copilot's suggested implementation for the character selection screen.

The suggestion was quite accurate and provided a solid starting point, allowing me to quickly develop the screen. However, Copilot did not provide a correct implementation for adding a hover transition to the buttons of the screen. This was somewhat expected, given that Ren'Py is not a widely used language.

Using AI tools for image generation

I wanted to create a simple game quickly, so I didn't have the time to design all the graphics myself. I used AI-powered image generation tools to create the characters and background images used in the game. My key requirements were consistent style across all images and a festive theme.

I started with the DALL-E 3 model, available through the Azure OpenAI Service. While the images were decent, I found it challenging to keep a consistent style across all images.

Then, I switched to Microsoft Designer, an AI-powered cloud-based graphic design application by Microsoft. This gave me more consistent results across different requests, making it a better fit for my needs. For character creation, I used the avatar text-to-image feature, and for the backgrounds, I used the default one. After experimenting with several styles, I settled on a "low poly" aesthetic because it delivered both consistent results and a look that matched the game's playful, gamified nature. Here's an example of a prompt I used to generate a character image:

"Low-poly 3D portrait of a stylized woman with brown hair, wearing a blouse in christmas colors, featuring clean geometric shapes, flat colors, and soft lighting, in a minimalist futuristic style with white background."

The generated images were processed by applying filters to reduce noise, smooth the colors, remove the background, and highlight the edges of the polygons. In some cases, I combined two images to create the final character design and generated duplicates with closed eyes to produce the eye blinking effect in the game.

Overall, I found these tools extremely helpful for quickly generating visual assets. While I created hundreds of versions for each character (okay, maybe I'm a bit of a perfectionist!), AI tools were invaluable for generating images that aligned with the game's theme.

Summary

In this post, we explored how Ren'Py, a visual novel engine built on Python, and AI tools such as GitHub Copilot, Azure OpenAI Service, and Microsoft Designer can be used to create a prototype for an educational game. But what's next? Now, we need to deploy the app. Fortunately, Ren'Py offers a CLI tool to automate game builds, and Azure Static Web Apps integrates seamlessly with GitHub Actions. In the next post, I'll show you how I set up a GitHub Actions workflow to automate the build and deployment process for the game.

In the meantime, you can check out the following resources to learn more about GitHub Copilot and the DALL-E 3 model:

Use HNSW index on Azure Cosmos DB for PostgreSQL for similarity search

Foteini Savvidou — Thu, 14 Mar 2024 20:45:02 +0000

In the previous post, you explored the IVFFlat (Inverted File with Flat Compression) index for approximate nearest neighbor search on Azure Cosmos DB for PostgreSQL. You observed that the IVFFlat index provides accurate results with lower search times compared to exact nearest neighbor search.

The pgvector extension provides another indexing algorithm for approximate nearest neighbor search called Hierarchical Navigable Small World (HNSW) graphs. HNSW is one of the most popular and best-performing indexes for vector similarity search. HNSW index support was introduced in pgvector 0.5.0.

In this tutorial, you will:

Create an HNSW index in your Azure Cosmos DB for PostgreSQL table.
Write SQL queries to detect similar images based on a text prompt or a reference image, utilizing the HNSW index.
Investigate the execution plan of a similarity search query.

Prerequisites

To proceed with this tutorial, ensure that you have the following prerequisites installed and configured:

An Azure subscription - Create an Azure free account or an Azure for Students account.
Python 3.10, Visual Studio Code, Jupyter Notebook, and Jupyter Extension for Visual Studio Code.

Set-up your working environment

In this guide, you'll learn how to query embeddings stored in an Azure Cosmos DB for PostgreSQL table to search for images similar to a search term or a reference image. The entire functional project is available in my GitHub repository. If want to follow along, just fork the repository and clone it to have it locally available.

Before running the Jupyter Notebook covered in this post, you should:

Create a virtual environment and activate it.
Install the required Python packages using the following command:
```
pip install -r requirements.txt
```
Create vector embeddings for a collection of images by running the scripts found in the data_processing directory.
Upload the images to your Azure Blob Storage container by executing the script found in the data_upload directory.

How the HNSW index works

The HNSW index is based on the construction of a multi-layered graph structure, that is optimized for performing approximate nearest neighbor search. In this graph structure, the datapoints (also referred to as nodes or vertices) are connected to each other by edges, which make it possible to navigate through the graph by following these edges.

The base layer of the multi-layer graph essentially represents the entire dataset, while the higher layers consist of fewer nodes, providing a simplified overview of the layers below. The higher layers contain longer links, allowing for longer jumps between nodes for faster search, while the lower layers contain shorter links, enabling more accurate search. Nearest neighbor search begins at the top layer, where the longest links are present. We then navigate to the nearest node and gradually move to lower layers until a local minimum is found. This process is illustrated in the following image:

The search process in an HNSW graph.

The search process in an HNSW graph can be compared to the process of planning a trip between two cities. Much like how we start our journey with major roads and gradually transition to smaller ones as we approach our destination, the HNSW search process begins with longer links at the top layer and gradually moves to lower layers as we approach the desired data points.

The HNSW algorithm is based on two fundamentals techniques: the probability skip list and the navigable small world (NSW) graphs. A detailed explanation of the process of constructing an index and searching through the graph is beyond the scope of this article. For further information, refer to the resources provided at the end of this article.

Compared to the IVFFlat index, the HNSW index generally provides better query performance in terms of the tradeoff between recall and speed, but at the expense of higher build time and more memory usage. Additionally, it doesn't require a training step to build the index. This means you can create an HNSW index even before any data is inserted into the table, unlike the IVFFlat index, which needs to be rebuilt when data changes to accurately represent new cluster centroids.

Create an HNSW index

The code for creating an HNSW index and inserting data into a PostgreSQL table can be found at data_upload/upload_data_to_postgresql_hnsw.py.

To create an HNSW index through the pgvector extension, three parameters need to be specified:

Distance: The pgvector extension provides 3 methods for calculating the distance between vectors: Euclidean (L2), inner product, and cosine. These methods are identified by vector_l2_ops, vector_ip_ops, and vector_cosine_ops, respectively. It is essential to select the same distance metric for both the creation and querying of the index.
m: The parameter m specifies the maximum number of connections with neighboring datapoints per point per layer. Its default value is 16.
ef_construction: The parameter ef_construction defines the size of list that holds the nearest neighbor candidates when building the index. The default value is 64.

To create an HNSW index in a PostgreSQL table, you can use the following statement:

CREATE INDEX ON <table_name> USING hnsw (<vector_column_name> <distance_method>) WITH (m = <m>, ef_construction = <ef_construction>);

Detect similar images using the pgvector HNSW index

The code for image similarity search with the pgvector extension can be found at vector_search_samples/image_search_hnsw_index.ipynb.

To search for similar images through the HNSW index of the pgvector extension, we can use SQL SELECT statements and the built-in distance operators. The structure of a SELECT statement was explained in the Exact Nearest Neighbor Search blog post. For approximate nearest neighbor search, an additional parameter needs to be considered to use the HNSW index.

Approximate nearest neighbor search using the HNSW index

The ef_search parameter specifies the size of the list that holds the nearest neighbor candidates during query execution. The default value is set to 40. The ef_search parameter can be altered (for a single query or a session) using the following command:

SET hnsw.ef_search = <value>;

To check whether PostgreSQL utilizes the index in a query, you can prefix the SELECT statement with the EXPLAIN ANALYZE keywords. An example of a query plan that utilizes the HNSW index is provided below:

Limit (cost=160.60..163.02 rows=12 width=72) (actual time=1.283..1.406 rows=12 loops=1)
  ->  Index Scan using paintings_hnsw_vector_idx on paintings (cost=160.60..2416.67 rows=11206 width=72) (actual time=1.281..1.403 rows=12 loops=1)
        Order By: (vector <=> '[0.001363333, ..., -0.0010466448]'::vector)
Planning Time: 0.183 ms
Execution Time: 1.439 ms

Additionally, it is important to note that pgvector only supports ascending-order index scans. This means that the following query does not utilize the HNSW index:

SELECT image_title, artist_name, 1 - (vector <=> '[0.003, …, 0.034]') AS cosine_similarity FROM paintings ORDER BY cosine_similarity DESC LIMIT 12;

One possible way to rewrite the SELECT statement to use the index is provided below:

SELECT image_title, artist_name, vector <=> '[0.003, …, 0.034]' AS cosine_distance FROM paintings ORDER BY cosine_distance LIMIT 12;

Code sample: Image similarity search with HNSW index

In the Jupyter Notebook provided on my GitHub repository, you'll explore text-to-image and image-to-image search scenarios. You will use the same text prompts and reference images as in the Exact Nearest Neighbors search example, allowing for a comparison of the accuracy of the results.

Images retrieved by searching for paintings using the painting "Still Life with Flowers" by Charles Ginner as a reference. The HNSW index retrieved all the paintings obtained with exact search.

Feel free to experiment with the notebook and modify the code to gain hands-on experience with the pgvector extension!

Next steps

The pgvector extension and PostgreSQL provide additional features that you can leverage to build AI-powered search applications. For example, you can integrate vector search with conventional keyword-based search methods into hybrid search systems, which generally have better performance.

If you want to learn more about the HNSW algorithm, check out these learning resources:

Alexander Ponomarenko, Yury Malkov, Andrey Logvinov, Vladimir Krylov, Approximate Nearest Neighbor Search Small World Approach (2011)
Yury Malkov, Alexander Ponomarenko, Andrey Logvinov, Vladimir Krylov, Approximate nearest neighbor algorithm based on navigable small world graphs (2014)
Yury Malkov, Dmitry Yashunin, Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs (2020)
Vector Similarity Search and Faiss Course by James Briggs
Similarity Search, Part 4: Hierarchical Navigable Small World (HNSW) by Vyacheslav Efimov – Towards Data Science
How to optimize performance when using pgvector on Azure Cosmos DB for PostgreSQL – Microsoft Docs
Official GitHub repository of the pgvector extension

👋 Hi, I am Foteini Savvidou!
An Electrical and Computer Engineer and Microsoft AI MVP (Most Valuable Professional) from Greece.

🌈 LinkedIn | Blog | GitHub

Use IVFFlat index on Azure Cosmos DB for PostgreSQL for similarity search

Foteini Savvidou — Sun, 03 Mar 2024 11:45:39 +0000

In the previous post, you developed an image similarity search app using Jupyter Notebook. Utilizing the pgvector extension on Azure Cosmos DB for PostgreSQL, you were able to detect images that are semantically similar to a reference image or a text prompt.

By default, pgvector performs exact nearest neighbor search, calculating the similarity between the query vector and every vector in the database. While this type of search provides perfect recall, it often leads to longer search times. To enhance efficiency for large datasets, you should create indexes to enable approximate nearest neighbor search, which trades off result quality for speed. Pgvector supports two types of approximate indexes:

Inverted File with Flat Compression (IVFFlat) index
Hierarchical Navigable Small World (HNSW) index

In this article, we will explore similarity search using an IVFFlat index. In the next post, we will work with the HNSW index, which is one of the best performing indexes for vector similarity search.

In this tutorial, you will:

Create an IVFFlat index in your Azure Cosmos DB for PostgreSQL table.
Write SQL queries to detect similar images based on a text prompt or a reference image, utilizing the IVFFlat index.
Investigate the execution plan of a similarity search query.

Prerequisites

To proceed with this tutorial, ensure that you have the following prerequisites installed and configured:

An Azure subscription - Create an Azure free account or an Azure for Students account.
Python 3.10, Visual Studio Code, Jupyter Notebook, and Jupyter Extension for Visual Studio Code.

Set-up your working environment

In this guide, you'll learn how to query embeddings stored in an Azure Cosmos DB for PostgreSQL table to search for images similar to a search term or a reference image. The entire functional project is available in the GitHub repository. If want to follow along, just fork the repository and clone it to have it locally available.

Before running the Jupyter Notebook covered in this post, you should:

Create a virtual environment and activate it.
Install the required Python packages using the following command:
```
pip install -r requirements.txt
```
Create vector embeddings for a collection of images by running the scripts found in the data_processing directory.
Upload the images to your Azure Blob Storage container by executing the script found in the data_upload directory.

How the IVFFlat index works

The IVFFlat algorithm accelerates vector search by grouping the vectors in the dataset into clusters (also known as Voronoi regions or cells) and limiting the search scope to the few nearest clusters for each query rather than the entire dataset.

Let’s gain an intuitive understanding of how IVFFlat works. Consider that we place our high-dimensional vectors in a two-dimensional vector space. We then apply k-means clustering to compute the cluster centroids. After identifying the centroids, we assign each vector in our dataset to the closest centroid based on proximity. This process results in the partition of the vector space into several non-intersecting regions (Voronoi Diagram), as depicted in the following image:

The process of constructing the Voronoi diagram. In this scenario, we have four centroids, resulting in four Voronoi cells. Each vector is assigned to its closest centroid.

Now, each vector falls within a region. In the context of similarity search, each region consists of vectors that are semantically similar. Since vectors within the same region are more likely to be similar to each other than those in different regions, IVFFlat makes the search process more efficient.

Let’s consider the query vector, as shown in the following image. To find the nearest neighbors to this query vector using IVFFlat, we perform the following steps:

Calculate the distance between the query vector and each centroid.
Identify the cell whose centroid is closest to the query vector and limit the search scope to that cell.
Compute the distance between the query vector and every vector within the selected cell.
Choose the vectors with the smallest distance as the nearest neighbors to the query vector.

When using the IVFFlat index, errors may occur when searching for nearest neighbors to a vector located at the edge of two regions in the vector space. In this scenario, by considering only one cell during the search, only the orange region is examined. Despite the query vector being close to a datapoint in the blue region, these vectors will not be compared.

Although the IVFFlat algorithm accelerates the search process and provides good search quality, it can lead to errors. For example, the above image illustrates the scenario where the query vector resides at the edge of two cells. Despite the query vector being close to a datapoint in the blue region, this vector will not be considered a nearest neighbor candidate since the search scope is limited to the orange region.

To address this issue and enhance search quality, we can expand the search scope by selecting several regions to search for nearest neighbor candidates. However, this approach comes with a trade-off: it increases search time.

Create an IVFFlat index

The code for creating an IVFFlat index and inserting data into a PostgreSQL table can be found at data_upload/upload_data_to_postgresql_ivfflat.py.

To create an IVFFlat index using pgvector, two parameters need to be specified:

Distance: The pgvector extension provides 3 methods for calculating the distance between vectors: Euclidean (L2), inner product, and cosine. These methods are identified by vector_l2_ops, vector_ip_ops, and vector_cosine_ops, respectively. It is essential to select the same distance metric for both the creation and querying of the index.
Number of clusters: The lists parameter specifies the number of clusters that will be created. Pgvector suggests that an appropriate number of lists is rows/1000 for datasets with up to 1 million rows and sqrt(rows) for larger datasets. It is also advisable to create at least 10 clusters.

To create an IVFFlat index in a PostgreSQL table, you can use the following statement:

CREATE INDEX ON <table_name> USING ivfflat (<vector_column_name> <distance_method>) WITH (lists = <number_of_clusters>);

It is important to note that the index should be created once the table is populated with data.

If new vectors are added to the table, the index should be rebuilt to update the cluster centroids and accurately represent the new dataset.

Detect similar images using the pgvector IVFFlat index

The code for image similarity search with the pgvector extension can be found at vector_search_samples/image_search_ivfflat_index.ipynb.

To search for similar images through the IVFFlat index of the pgvector extension, we can use SQL SELECT statements and the built-in distance operators. The structure of a SELECT statement was explained in the Exact Nearest Neighbor Search blog post. For approximate nearest neighbor search, some additional parameters need to be considered to use the IVFFlat index.

Approximate nearest neighbor search using the IVFFlat index

The number of regions to consider during search is determined by the probes parameter. According to pgvector documentation, the recommended value for the probes parameter is sqrt(lists). The default value is 1. To specify the value of the probes parameter, you can execute the following statement:

SET ivfflat.probes = <number_of_probes>;

To specify the number of probes to use for a single query, you should use the LOCAL keyword.

It's important to note that PostgreSQL does not guarantee the use of an approximate index, as it may determine that a sequential scan could be more efficient for a query. To check whether PostgreSQL utilizes the index in a query, you can prefix the SELECT statement with the EXPLAIN ANALYZE keywords.

EXPLAIN ANALYZE SELECT image_title, artist_name FROM paintings ORDER BY vector <=> '[0.001363333, …, -0.0010466448]' LIMIT 12;

An example of a query plan that utilizes the IVFFlat index is provided below:

Limit  (cost=1537.64..1539.15 rows=12 width=72) (actual time=1.803..1.984 rows=12 loops=1)
  ->  Index Scan using paintings_ivfflat_vector_idx on paintings (cost=1537.64..2951.85 rows=11206 width=72) (actual time=1.801..1.981 rows=12 loops=1)
        Order By: (vector <=> '[0.001363333, ..., -0.0010466448]'::vector)
Planning Time: 0.135 ms
Execution Time: 2.014 ms

Additionally, you may need to rewrite (or simplify) your queries in order to use an approximate index. For example, since pgvector only supports ascending-order index scans, the following query does not utilize the IVFFlat index:

SELECT image_title, artist_name, 1 - (vector <=> '[0.003, …, 0.034]') AS cosine_similarity FROM paintings ORDER BY cosine_similarity DESC LIMIT 12;

If we want to utilize the index, one possible way to rewrite the SELECT statement is as follows:

SELECT image_title, artist_name, vector <=> '[0.003, …, 0.034]' AS cosine_distance FROM paintings ORDER BY cosine_distance LIMIT 12;

Code sample: Image similarity search with IVFFlat index

Images retrieved by searching for paintings using the painting "Still Life with Flowers" by Charles Ginner as a reference. The IVFFlat index with probes=1 retrieved 9 out of the 12 paintings obtained with exact search.

Feel free to experiment with the notebook and modify the code to gain hands-on experience with the pgvector extension!

Next steps

In this post, you used the IVFFlat indexing algorithm of the pgvector extension to search for paintings that closely match a reference image or a text prompt. In the upcoming post, we will explore the workings of the HNSW index and use it for similarity searches.

If you want to explore pgvector's features, check out these learning resources:

👋 Hi, I am Foteini Savvidou!
An Electrical and Computer Engineer and Microsoft AI MVP (Most Valuable Professional) from Greece.

🌈 LinkedIn | Blog | GitHub

Use pgvector for searching images on Azure Cosmos DB for PostgreSQL

Foteini Savvidou — Wed, 07 Feb 2024 18:28:48 +0000

Welcome to the next part of the "Image similarity search with pgvector" learning series!

In the previous articles, you used the multi-modal embeddings APIs of Azure AI Vision for generating embeddings for a collection of images of paintings and stored the embeddings in an Azure Cosmos DB for PostgreSQL table.

Introduction

If you have followed the previous posts, you should have successfully created a table in your Azure Cosmos DB for PostgreSQL cluster, populated it with data, and uploaded the images to a container in your Azure Storage account. Now, you are fully prepared to search for similar images utilizing the vector similarity search features of the pgvector extension.

In this tutorial, you will:

Develop an image similarity search app using Jupyter Notebook.
Write SQL queries to detect similar images based on a text prompt or a reference image.
Apply a simple metadata filtering method to narrow down search results.

Prerequisites

To proceed with this tutorial, ensure that you have the following prerequisites installed and configured:

An Azure subscription - Create an Azure free account or an Azure for Students account.
Python 3.10, Visual Studio Code, Jupyter Notebook, and Jupyter Extension for Visual Studio Code.

Set-up your working environment

In this guide, you'll learn how to query embeddings stored in an Azure Cosmos DB for PostgreSQL table to search for images similar to a search term or a reference image. The entire functional project is available in the GitHub repository. If you're keen on trying it out, just fork the repository and clone it to have it locally available.

Before running the Jupyter Notebook covered in this post, you should:

Create a virtual environment and activate it.
Install the required Python packages using the following command:
```
pip install -r requirements.txt
```
Create vector embeddings for a collection of images by running the scripts found in the data_processing directory.
Upload the images to your Azure Blob Storage container, create a PostgreSQL table, and populate it with data by executing the scripts found in the data_upload directory.

Detect similar images using the pgvector extension

The code for image similarity search with the pgvector extension can be found at vector_search_samples/image_search.ipynb.

The image similarity search workflow that we will follow is summarized as follows:

Use the Azure AI Vision Vectorize Image API or the Vectorize Text API to generate the vector embedding of a reference image or text prompt, respectively. It is crucial to employ the same embedding model for queries as the one used to generate embeddings for the images in the dataset.
To calculate similarity and retrieve images, utilize SQL SELECT statements and the built-in vector operators of the PostgreSQL database. Specifically, cosine similarity will be used as the similarity metric.
The similarity search will produce a list of vectors that are most similar to the query vector. The raw data associated with each vector can then be accessed.
Download the images from the Azure Blob Storage container and display them using the matplotlib package.

This workflow is illustrated in the following diagram:

Nearest neighbor search using pgvector

Given the vector embedding of the query, we can use SQL SELECT statements to search for similar images. Let’s understand how a simple SELECT statement works. Consider the following query:

SELECT * FROM table ORDER BY vector <=> '[0.003, …, 0.034]' LIMIT 5

This query computes the cosine distance (<=>) between the given vector ([0.003, …, 0.034]) and the vectors stored in the table, sorts the results by the calculated distance, and returns the five most similar images (LIMIT 5). Additionally, you can obtain the cosine similarity between the query vector and the retrieved vectors by modifying the SELECT statement as follows:

SELECT image_title, 1 - (vector <=> '[0.003, …, 0.034]') AS cosine_similarity FROM table ORDER BY cosine_similarity DESC LIMIT 5;

The pgvector extension provides 3 operators that can be used to calculate similarity:

Operator	Description
`<->`	Euclidean distance
`<#>`	Negative inner product
`<=>`	Cosine distance

Code sample: Image similarity search

In the Jupyter Notebook provided on my GitHub repository, you'll explore the following scenarios:

Text-to-image search: You will use a text prompt to search for and identify paintings that are semantically similar, relying solely on the vector embeddings without utilizing image metadata, such as the title or description of the painting.
Image-to-image search: You will use a painting as a reference to search for similar ones by comparing the vector embedding of the reference image with those in the collection.
Metadata filtering: Filtering enables users to narrow down search results, such as searching for paintings by a specific artist. However, implementing accurate and fast metadata filtering in vector search systems is a challenging task. You can read the article The Missing WHERE Clause in Vector Search on the Pinecone blog to learn about the two fundamental approaches for metadata filtering and understand the complexities involved in implementing such filters into vector search applications.

Images retrieved by searching for paintings using the text prompt "flowers by Vincent van Gogh".

Feel free to experiment with the notebook and modify the code to gain hands-on experience with the pgvector extension!

Next steps

In this post, you explored the basic vector similarity search features offered by the pgvector extension. This type of vector search is referred to as exact nearest neighbor search, as it computes the similarity between the query vector and every vector in the database. In the upcoming post, you will explore approximate nearest neighbor search, which trades off result quality for speed.

If you want to explore pgvector's features, check out these learning resources:

👋 Hi, I am Foteini Savvidou!
An Electrical and Computer Engineer and Microsoft AI MVP (Most Valuable Professional) from Greece.

🌈 LinkedIn | Blog | GitHub

Store embeddings in Azure Cosmos DB for PostgreSQL with pgvector

Foteini Savvidou — Mon, 29 Jan 2024 19:44:19 +0000

Welcome to the third part of the "Image similarity search with pgvector" learning series!

In the previous articles, you learned how to describe vector embeddings and vector similarity search. You also used the multi-modal embeddings APIs of Azure AI Vision for generating embeddings for a collection of images of paintings.

Introduction

In this learning series, we will create a search system that lets users provide a text description or a reference image to find similar paintings. We have already generated vector embeddings for the images in our dataset using the multi-modal embeddings API of Azure AI Vision. In this post, we will use Azure Blob Storage to store the images and Azure Cosmos DB for PostgreSQL to store our vector embeddings using the pgvector extension. In the next tutorials, we will perform a similarity search on our embeddings.

The workflow is illustrated in the following image:

In this tutorial, you will learn how to:

Upload images to an Azure Blob Storage container using the Python SDK.
Activate the pgvector extension on Azure Cosmos DB for PostgreSQL.
Store vector embeddings on an Azure Cosmos DB for PostgreSQL table.

Prerequisites

To proceed with this tutorial, ensure that you have the following prerequisites installed and configured:

An Azure subscription - Create an Azure free account or an Azure for Students account.
Python 3.10, Visual Studio Code, Jupyter Notebook, and Jupyter Extension for Visual Studio Code.

Set-up your working environment

In this guide, you'll learn how to upload a collection of paintings' images to an Azure Blob Storage container and insert vector embeddings into an Azure Cosmos DB for PostgreSQL table. The entire functional project is available in the GitHub repository. If you're keen on trying it out, just fork the repository and clone it to have it locally available.

Before running the scripts covered in this post, you should:

Create a virtual environment and activate it.
Install the required Python packages using the following command:
```
pip install -r requirements.txt
```
Create vector embeddings for a collection of images by running the scripts found in the data_processing directory.

Upload images to Azure Blob Storage

The code for uploading images to an Azure Blob Storage container can be found at data_upload/upload_images_to_blob.py.

Azure Blob Storage is a cloud storage service that is optimized for storing large amounts of unstructured data, such as images. It offers three types of resources:

The storage account that contains all your Azure Storage data objects. Every object that is stored in Azure Storage is identified by a unique address.
Containers in the storage account which are similar to directories in a file system.
Blobs that are organized in the containers.

The following diagram illustrates the relationship between these resources:

Azure Blob Storage resources: storage account, container, and blobs. Image source: Azure Blob Storage Object model – Microsoft Docs

Create an Azure Storage account

Open the Azure CLI.

Create an Azure Storage Account using the following command:

az storage account create --name account-name --resource-group your-group-name --location your-location --sku Standard_LRS --allow-blob-public-access true --min-tls-version TLS1_2

Upload images to an Azure Blob Storage container

The Azure Blob Storage client library for Python provides the following classes to manage blobs and containers:

BlobServiceClient: We will use the BlobServiceClient class to interact with the Azure Storage account and create a container.
ContainerClient: We will use the ContainerClient class to interact with our container and the blobs inside the container.
BlobClient: We will use the BlobClient class to upload a blob to our container.

The process of uploading our images to Azure Blob Storage can be summarized as follows:

Create a new container to store our images.
Retrieve the filenames of the images in the dataset.
Upload the images in the container, utilizing multiple threads via the ThreadPoolExecutor class. Additionally, use the tqdm library to display progress bars for better visualizing the image uploading process.

import os
import csv
import sys
from dotenv import load_dotenv
from azure.storage.blob import BlobServiceClient, ContainerClient, ContentSettings
from azure.core.exceptions import ResourceExistsError
from concurrent.futures import ThreadPoolExecutor
from tqdm import tqdm

# Constants
MAX_WORKERS = 4
IMAGE_FILE_CSV_COLUMN_NAME = "image_file"

# Directories
current_dir = os.path.dirname(os.path.realpath(__file__))
parent_dir = os.path.dirname(current_dir)

# Load environemt file
load_dotenv(os.path.join(parent_dir, ".env"), override=True)
# Azure Blob Storage credentials
blob_account_name = os.getenv("BLOB_ACCOUNT_NAME")
blob_account_key = os.getenv("BLOB_ACCOUNT_KEY")
blob_endpoint_suffix = os.getenv("BLOB_ENDPOINT_SUFFIX")
blob_connection_string = (
    f"DefaultEndpointsProtocol=https;AccountName={blob_account_name};"
    f"AccountKey={blob_account_key};EndpointSuffix={blob_endpoint_suffix}"
)
container_name = os.getenv("CONTAINER_NAME")

# Dataset's folder
dataset_folder = os.path.join(parent_dir, "dataset")
dataset_filepath = os.path.join(dataset_folder, "dataset_embeddings.csv")

# Images' folder
images_folder = os.path.join(parent_dir, "semart_dataset", "images")

# Content-Type for blobs
content_settings = ContentSettings(content_type="image/jpeg")


def main():
    # Create Azure Blob Storage client
    blob_service_client = BlobServiceClient.from_connection_string(conn_str=blob_connection_string)

    # Create a new container
    try:
        container_client = blob_service_client.create_container(name=container_name, public_access="blob")
    except ResourceExistsError:
        sys.exit(f"A container with name {container_name} already exists.")

    # Find the URLs of the images in the dataset
    images = load_image_filenames()

    print(f"Number of images in the dataset: {len(images)}")
    print(f"Uploading images to container '{container_name}'")

    # Upload images to blob storage
    upload_images(images=images, container_client=container_client)


def load_image_filenames() -> list[str]:
    with open(dataset_filepath, "r") as csv_file:
        csv_reader = csv.DictReader(csv_file, delimiter="\t", skipinitialspace=True)
        image_filenames = [row[IMAGE_FILE_CSV_COLUMN_NAME] for row in csv_reader]

    return image_filenames


def upload_blob_from_local_file(
    image_filepath: str,
    container_client: ContainerClient,
) -> None:
    blob_name = os.path.basename(image_filepath)
    try:
        blob_client = container_client.get_blob_client(blob=blob_name)
        with open(image_filepath, mode="rb") as data:
            blob_client.upload_blob(data=data, overwrite=True, content_settings=content_settings)
    except Exception as e:
        print(f"Couldn't upload image {blob_name} to Azure Storage Account due to error: {e}")


def upload_images(images: list[str], container_client: ContainerClient) -> None:
    with ThreadPoolExecutor(max_workers=MAX_WORKERS) as executor:
        list(
            tqdm(
                executor.map(
                    lambda x: upload_blob_from_local_file(
                        image_filepath=os.path.join(images_folder, x),
                        container_client=container_client,
                    ),
                    images,
                ),
                total=len(images),
            )
        )

Store embeddings in Azure Cosmos DB for PostgreSQL

The code for inserting vector embeddings into an Azure Cosmos DB for PostgreSQL table can be found at data_upload/upload_data_to_postgresql.py.

Create an Azure Cosmos DB for PostgreSQL cluster

Let's use the Azure portal to create an Azure Cosmos DB for PostgreSQL cluster.

Search for “Azure Cosmos DB for PostgreSQL” and then select +Create.
Fill out the information on the Basics tab:
- Subscription: Select your subscription.
- Resource group: Select your resource group.
- Cluster name: Enter a name for your Azure Cosmos DB for PostgreSQL cluster.
- Location: Choose your preferred region.
- Scale: You can leave Scale as its default value or select the optimal number of nodes as well as compute, memory, and storage configuration. Burstable, 1 vCores / 2 GiB RAM, 32 GiB storage is sufficient for this demo.
- PostgreSQL version: Choose a PostgreSQL version such as 15.
- Database name: You can leave database name at its default value citus.
- Administrator account: The admin username must be citus. Select a password that will be used for citus role to connect to the database.
On the Networking tab, select Allow public access from Azure services and resources within Azure to this cluster and create your preferred firewall rule.
Select Review + Create. Once the deployment is complete, navigate to your resource.

Activate the pgvector extension

The pgvector extension adds vector similarity search capabilities to your PostgreSQL database. To use the extension, you have to first create it in your database. You can install the extension, by connecting to your database and running the CREATE EXTENSION command from the psql command prompt:

SELECT CREATE_EXTENSION('vector');

The pgvector extension introduces a data type called VECTOR that can be used during the creation of a table to indicate that a column will hold vector embeddings. When creating the column, it's essential to specify the dimension of the vectors. In our scenario, Azure AI Vision generates 1024-dimensional vectors.

Insert data into a PostgreSQL table

To insert data into an Azure Cosmos DB for PostgreSQL table, we will proceed as follows:

Create a table to store the filenames of the images, their embeddings, and their associated metadata. All information is saved in a CSV file, as presented in Part 2.
Insert the data from the CSV file into the table using the PostgreSQL COPY command.

import os
import psycopg2
from psycopg2 import pool
from dotenv import load_dotenv

# Constants
IMAGE_FILE_COLUMN_NAME = "image_file"
DESCRIPTION_COLUMN_NAME = "description"
AUTHOR_COLUMN_NAME = "author"
TITLE_COLUMN_NAME = "title"
TECHNIQUE_COLUMN_NAME = "technique"
TYPE_COLUMN_NAME = "type"
TIMEFRAME_COLUMN_NAME = "timeframe"
VECTOR_COLUMN_NAME = "vector"

# Directories
current_dir = os.path.dirname(os.path.realpath(__file__))
parent_dir = os.path.dirname(current_dir)

# Load environemt file
load_dotenv(os.path.join(parent_dir, ".env"), override=True)
# Azure CosmosDB for PostgreSQL credentials
postgres_host = os.getenv("POSTGRES_HOST")
postgres_database_name = os.getenv("POSTGRES_DB_NAME")
postgres_user = os.getenv("POSTGRES_USER")
postgres_password = os.getenv("POSTGRES_PASSWORD")
sslmode = "require"
table_name = os.getenv("POSTGRES_TABLE_NAME")
postgres_connection_string = (
    f"host={postgres_host} user={postgres_user} "
    f"dbname={postgres_database_name} "
    f"password={postgres_password} sslmode={sslmode}"
)

# Dataset's folder
dataset_folder = os.path.join(parent_dir, "dataset")
dataset_filepath = os.path.join(dataset_folder, "dataset_embeddings.csv")


def main():
    postgresql_pool = psycopg2.pool.SimpleConnectionPool(1, 20, postgres_connection_string)
    if (postgresql_pool):
        print("Connection pool created successfully")

    # Get a connection from the connection pool
    conn = postgresql_pool.getconn()
    cursor = conn.cursor()

    print("Creating a table...")
    cursor.execute(f"DROP TABLE IF EXISTS {table_name};")
    cursor.execute(
        f"CREATE TABLE {table_name} ("
        f"{IMAGE_FILE_COLUMN_NAME} TEXT PRIMARY KEY,"
        f"{DESCRIPTION_COLUMN_NAME} TEXT NOT NULL,"
        f"{AUTHOR_COLUMN_NAME} TEXT NOT NULL,"
        f"{TITLE_COLUMN_NAME} TEXT NOT NULL,"
        f"{TECHNIQUE_COLUMN_NAME} TEXT,"
        f"{TYPE_COLUMN_NAME} TEXT,"
        f"{TIMEFRAME_COLUMN_NAME} TEXT,"
        f"{VECTOR_COLUMN_NAME} VECTOR(1024) NOT NULL);"
    )

    print("Saving data to table...")
    with open(dataset_filepath) as csv_file:
        cursor.copy_expert(
            f"COPY {table_name} FROM STDIN WITH "
            f"(FORMAT csv, DELIMITER '\t', HEADER MATCH);",
            csv_file
        )

    conn.commit()

    # Fetch all rows from table
    cursor.execute(f"SELECT * FROM {table_name};")
    rows = cursor.fetchall()
    print(f"Number of records in the table: {len(rows)}")

    # Close the connection
    cursor.close()
    conn.close()

Next steps

In this post, you uploaded the paintings’ images into an Azure Blob Storage container, configured the Azure Cosmos DB for PostgreSQL database as a vector database using the pgvector extension, and inserted the data into a table. In the subsequent posts, you will leverage the pgvector extension to perform a similarity search on the embeddings.

If you want to explore pgvector's features, check out these learning resources:

👋 Hi, I am Foteini Savvidou!
An Electrical and Computer Engineer and Microsoft AI MVP (Most Valuable Professional) from Greece.

🌈 LinkedIn | Blog | GitHub

Generate embeddings with Azure AI Vision multi-modal embeddings API

Foteini Savvidou — Mon, 22 Jan 2024 18:11:06 +0000

Welcome to the second part of the “Image similarity search with pgvector” learning series! In the previous article, you learned how to describe vector embeddings and vector similarity search. You also used the multi-modal embeddings APIs of Azure AI Vision for generating embeddings for images and text and calculated the cosine similarity between two vectors.

Introduction

In this learning series, we will create an application that enables users to search for paintings based on either a reference image or a text description. We will use the SemArt Dataset, which contains approximately 21k paintings gathered from the Web Gallery of Art. Each painting comes with various attributes, like a title, description, and the name of the artist.

In this tutorial, you will:

Prepare the data for further processing.
Generate vector embeddings for a collection of images of paintings using the Vectorize Image API of Azure AI Vision.

Prerequisites

To proceed with this tutorial, ensure that you have the following prerequisites installed and configured:

An Azure subscription - Create an Azure free account or an Azure for Students account.
An Azure AI Vision resource - For instructions on creating an Azure AI Vision resource, see Part 1.
Python 3.10, Visual Studio Code, Jupyter Notebook, and Jupyter Extension for Visual Studio Code.

Set-up your working environment

In this article, you will find instructions on how to generate embeddings for a collection of images using Azure AI Vision. The complete working project can be found in the GitHub repository. If you want to follow along, you can fork the repository and clone it to have it locally available.

Before running the scripts, you should:

Download the SemArt Dataset into the semart_dataset directory.
Create a virtual environment and activate it.
Install the required Python packages using the following command:
```
pip install -r requirements.txt
```

Data preprocessing

Note: The code for data preprocessing can be found at data_processing/data_preprocessing.ipynb.

For our application, we'll be working with a subset of the original dataset. Alongside the image files, we aim to retain associated metadata like the title, author's name, and description for each painting. To prepare the data for further processing and eliminate unnecessary information, we will take several steps as outlined in the Jupyter Notebook available on my GitHub repository:

Clean up the text descriptions by removing special characters to minimize errors related to character encoding.
Clean up the names of the artists, addressing encoding issues for some artists' names.
Exclude artists with fewer than 15 paintings from the dataset, along with other data we won't be using.

After these steps, the final dataset will comprise 11,206 images of paintings.

Create vector embeddings with Azure AI Vision

Note: The code for vector embeddings generation can be found at data_processing/generate_embeddings.py.

To generate embeddings for the images, our process can be summarized as follows:

Retrieve the filenames of the images in the dataset.
Divide the data into batches, and for each batch, perform the following steps:
1. Compute the vector embedding for each image in the batch using the Vectorize Image API of Azure AI Vision.
2. Save the vector embeddings of the images along with the filenames into a file.
Update the dataset by inserting the vector embedding of each image.

In the following sections, we will discuss specific segments of the code.

Compute embeddings for the images in the dataset

As discussed in Part 1, computing the vector embedding of an image involves sending a POST request to the Azure AI Vision retrieval:vectorizeImage API. The binary image data (or a publicly available image URL) is included in the request body, and the response consists of a JSON object containing the vector embedding of the image. In Python, this can be achieved by utilizing the requests library to send a POST request.

def get_image_embedding(image: str) -> list[float] | None:
    """
    Generates a vector embedding for an image using Azure AI Vision 4.0
    (Vectorize Image API).

    :param image: The image filepath.
    :return: The vector embedding of the image.
    """
    with open(image, "rb") as img:
        data = img.read()

    headers = {
        "Content-type": "application/octet-stream",
        "Ocp-Apim-Subscription-Key": vision_key,
    }

    try:
        r = requests.post(vectorize_img_url, data=data, headers=headers)
        if r.status_code == 200:
            image_vector = r.json()["vector"]
            return image_vector
        else:
            print(
                f"An error occurred while processing {image}. "
                f"Error code: {r.status_code}."
            )
    except Exception as e:
        print(f"An error occurred while processing {image}: {e}")

    return None

The compute_embeddings function computes the vector embeddings for all the images in our dataset. It uses the ThreadPoolExecutor object to generate vector embeddings for each batch of images efficiently, utilizing multiple threads. The tqdm library is also utilized in order to provide progress bars for better visualizing the embeddings generation process.

def compute_embeddings(image_names: list[str]) -> None:
    """
    Computes vector embeddings for the provided images and saves the embeddings
    alongside their corresponding image filenames in a CSV file.

    :param image_names: A list containing the filenames of the images.
    """
    image_names_batches = [
        image_names[i:(i + BATCH_SIZE)]
        for i in range(0, len(image_names), BATCH_SIZE)
    ]
    for batch in tqdm(range(len(image_names_batches)), desc="Computing embeddings"):
        images = image_names_batches[batch]
        with ThreadPoolExecutor(max_workers=MAX_WORKERS) as executor:
            embeddings = list(
                tqdm(
                    executor.map(
                        lambda x: get_image_embedding(
                            image=os.path.join(images_folder, x),
                        ),
                        images,
                    ),
                    total=len(images),
                    desc=f"Processing batch {batch+1}",
                    leave=False,
                )
            )
        valid_data = [
            [images[i], str(embeddings[i])] for i in range(len(images))
            if embeddings[i] is not None
        ]
        save_data_to_csv(valid_data)

Once the embeddings for all the images in a batch are computed, the data is saved into a CSV file.

def save_data_to_csv(data: list[list[str]]) -> None:
    """
    Appends a list of image filenames and their associated embeddings to
    a CSV file.

    :param data: The data to be appended to the CSV file.
    """
    with open(embeddings_filepath, "a", newline="") as csv_file:
        write = csv.writer(csv_file)
        write.writerows(data)

Azure AI Vision API rate limits

Azure AI Vision API imposes rate limits on its usage. In the free tier, only 20 transactions per minute are allowed, while the standard tier allows up to 30 transactions per second, depending on the operation (Source: Microsoft Docs). If you exceed the default rate limit, you'll receive a 429 HTTP error code.

For our application, it is recommended to use the standard tier during the embeddings generation process and limit the number of requests per second to approximately 10 to avoid potential issues.

Generate the dataset

After computing the vector embeddings for all images in the dataset, we proceed to update our dataset by inserting the vector embedding for each image. In the generate_dataset function, the merge method of pandas.DataFrame is used for merging the dataset with a database-style join.

def generate_dataset() -> None:
    """
    Appends the corresponding vectors to each column of the original dataset
    and saves the updated dataset as a CSV file.
    """
    dataset_df = pd.read_csv(dataset_filepath, sep="\t", dtype="string")
    embeddings_df = pd.read_csv(
        embeddings_filepath,
        dtype="string",
        names=[IMAGE_FILE_CSV_COLUMN_NAME, EMBEDDINGS_CSV_COLUMN_NAME],
    )
    final_dataset_df = dataset_df.merge(
        embeddings_df, how="inner", on=IMAGE_FILE_CSV_COLUMN_NAME
    )
    final_dataset_df.to_csv(final_dataset_filepath, index=False, sep="\t")

Next steps

In this post, we computed vector embeddings for a set of images featuring paintings using the Azure AI Vision Vectorize Image API. The code shared here serves as a reference, and you can customize it to suit your particular use case.

Here are some additional learning resources:

👋 Hi, I am Foteini Savvidou!
An Electrical and Computer Engineer and Microsoft AI MVP (Most Valuable Professional) from Greece.

🌈 LinkedIn | Blog | GitHub

Use the Azure AI Vision multi-modal embeddings API for image retrieval

Foteini Savvidou — Mon, 22 Jan 2024 17:58:04 +0000

Welcome to a new learning series about image similarity search with pgvector, an open-source vector similarity search extension for PostgreSQL databases.

I find vector search an intriguing technology, and I’ve decided to explore it! Throughout this series, I will discuss the basic concepts of vector search, introduce you to the multi-modal embeddings API of Azure AI Vision, and guide you in building an image similarity search application using Azure Cosmos DB for PostgreSQL.

Introduction

Conventional search systems rely on exact matches on properties like keywords, tags, or other metadata, lexical similarity, or the frequency of word occurrences to retrieve similar items. Recently, vector similarity search has transformed the search process. It leverages machine learning to capture the meaning of data, allowing you to find similar items based on their content. The key idea behind vector search involves converting unstructured data, such as text, images, videos, and audio, into high-dimensional vectors (also known as embeddings) and applying nearest neighbor algorithms to find similar data.

In this tutorial, you learn how to:

Describe vector embeddings and vector similarity search.
Use the multi-modal embeddings API of Azure AI Vision for generating vectors for images and text.

Prerequisites

To proceed with this tutorial, ensure that you have the following prerequisites installed and configured:

An Azure subscription - Create an Azure free account or an Azure for Students account.
Python 3.x, Visual Studio Code, Jupyter Notebook, and Jupyter Extension for Visual Studio Code.

Concepts

Vector embeddings

Comparing unstructured data is challenging, in contrast to numerical and structured data, which can be easily compared by performing mathematical operations. What if we could convert unstructured data, such as text and images, into a numerical representation? We could then calculate their similarity using standard mathematical methods.

These numerical representations are called vector embeddings. An embedding is a high-dimensional and dense vector that summarizes the information contained in the original data. Vector embeddings can be computed using machine learning algorithms that capture the meaning of the data, recognize patterns, and identify similarities between the data.

Visualization of word embeddings in a 2-dimensional vector space. Words that are semantically similar are located close together, while dissimilar words are placed farther apart.

Vector similarity

The numerical distance between two embeddings, or equivalently, their proximity in the vector space, represents their similarity. Vector similarity is commonly calculated using distance metrics such as Euclidean distance, inner product, or cosine distance.

Cosine is the similarity metric used by Azure AI Vision. This metric measures the angle between two vectors and is not affected by their magnitudes. Mathematically, cosine similarity is defined as the cosine of the angle between two vectors, which is equal to the dot product of the vectors divided by the product of their magnitudes.

Vector similarity can be used in various industry applications, including recommender systems, fraud detection, text classification, and image recognition. For example, systems can use vector similarities between products to identify similar products and create recommendations based on a user's preferences.

Vector similarity search

A vector search system works by comparing the vector embedding of a user’s query with a set of pre-stored vector embeddings to find a list of vectors that are the most similar to the query vector. The diagram below illustrates this workflow.

Overview of vector similarity search flow.

Vector embeddings are usually stored in a vector database, which is a specialized type of database that is optimized for storing and querying vectors with a large number of dimensions. You will learn more about vector databases in one of the following posts in this series.

Create vector embeddings with Azure AI Vision

Azure AI Vision provides two APIs for vectorizing image and text queries: the Vectorize Image API and the Vectorize Text API. This vectorization converts images and text into coordinates in a 1024-dimensional vector space, enabling users to search a collection of images using text and/or images without the need for metadata, such as image tags, labels, or captions.

Let's learn how the multi-modal embeddings APIs work.

Create an Azure AI Vision resource

Open the Azure CLI.

Create a resource group using the following command:

az group create --name your-group-name --location your-location

Create an Azure AI Vision in the resource group that you have created using the following command:

az cognitiveservices account create --name ai-vision-resource-name --resource-group your-group-name --kind ComputerVision --sku S1 --location your-location --yes

Note: The multi-modal embeddings APIs are available in the following regions: East US, France Central, Korea Central, North Europe, Southeast Asia, West Europe, West US.

Before using the multi-modal embeddings APIs, you need to store the key and the endpoint of your Azure AI Vision resource in an environment (.env) file.

Use the Vectorize Image API

Let's review the following example. Given the filename of an image, the get_image_embedding function sends a POST API call to the retrieval:vectorizeImage API. The binary image data is included in the HTTP request body. The API call returns a JSON object containing the vector embedding of the image.

import os
from dotenv import load_dotenv
import requests

# Load environment variables
load_dotenv()
endpoint = os.getenv("VISION_ENDPOINT") + "computervision/"
key = os.getenv("VISION_KEY")

def get_image_embedding(image):
    with open(image, "rb") as img:
        data = img.read()

    # Vectorize Image API
    version = "?api-version=2023-02-01-preview&modelVersion=latest"
    vectorize_img_url = endpoint + "retrieval:vectorizeImage" + version

    headers = {
        "Content-type": "application/octet-stream",
        "Ocp-Apim-Subscription-Key": key
    }

    try:
        r = requests.post(vectorize_img_url, data=data, headers=headers)

        if r.status_code == 200:
            image_vector = r.json()["vector"]
            return image_vector
        else:
            print(f"An error occurred while processing {image}. Error code: {r.status_code}.")

    except Exception as e:
        print(f"An error occurred while processing {image}: {e}")

    return None

image_filename = "images/image (1).jpg"
image_vector = get_image_embedding(image_filename)

Info: To vectorize a remote image, you would put the URL of the image in the request body.

Use the Vectorize Text API

Similarly to the example above, the get_text_embedding function sends a POST API call to the retrieval:vectorizeText API.

import json

def get_text_embedding(prompt):
    text = {'text': prompt}

    # Image retrieval API
    version = "?api-version=2023-02-01-preview&modelVersion=latest"
    vectorize_txt_url = endpoint + "retrieval:vectorizeText" + version

    headers = {
        'Content-type': 'application/json',
        'Ocp-Apim-Subscription-Key': key
    }

    try:
        r = requests.post(vectorize_txt_url, data=json.dumps(text), headers=headers)

        if r.status_code == 200:
            text_vector = r.json()['vector']
            return text_vector
        else:
            print(f"An error occurred while processing the prompt '{text}'. Error code: {r.status_code}.")

    except Exception as e:
        print(f"An error occurred while processing the prompt '{text}': {e}")

    return None

text_prompt = "a blue house"
text_vector = get_text_embedding(text_prompt)

Calculate image similarity

The following code calculates the cosine similarity between the vector of the image and the vector of the text prompt.

from numpy import dot
from numpy.linalg import norm

def get_cosine_similarity(vector1, vector2):
    return dot(vector1, vector2) / (norm(vector1) * norm(vector2))

similarity = get_cosine_similarity(image_vector, text_vector)
print(similarity)

Next steps

In this article, you’ve learned the basics of vector search and explored the multi-modal embeddings API of Azure AI Vision. In the next post, you will create vector embeddings for a collection of paintings' images.

Here are some helpful learning resources:

👋 Hi, I am Foteini Savvidou!
An Electrical and Computer Engineer and Microsoft AI MVP (Most Valuable Professional) from Greece.

🌈 LinkedIn | Blog | GitHub

Building a vector similarity search app with Azure AI Vision and PostgreSQL

Foteini Savvidou — Sun, 05 Nov 2023 11:22:51 +0000

Have you ever found yourself in a situation where you remember what happened in a movie but struggle to recall its name and can’t come up with the right words to search for it? Or, maybe you’ve had that moment when you have a picture of a product you wish to purchase but aren't sure how to find it online.

Unlike traditional search systems that rely on mentions of keywords, tags, or other metadata, vector search leverages machine learning to capture the meaning of data, allowing you to search by what you mean. Consider the second scenario from the paragraph above as an example. In a vector search system, you can simply use the image of a product to find similar ones instead of struggling to come up with the proper search words. This method is also known as "reverse image search."

Vector search works by transforming data, such as text, images, videos, and audio, into a numerical representation that is called vector embedding and applying nearest neighbor algorithms to find similar data.

Introduction

A few days ago, I had a wonderful time delivering a session about building a vector search system using Azure Cosmos DB for PostgreSQL and Azure AI Vision at the monthly Azure Cosmos DB Usergroup’s virtual show hosted by Jay Gordon and Microsoft Reactor. In this article, I will document the main points of my session and provide you with learning resources and code examples.

You will:

Understand the concept of vector embeddings and how a vector search similarity system works.
Use the Azure AI Vision Image Retrieval APIs for converting images and text into a vector representation.
Build a vector search system with Azure Cosmos DB for PostgreSQL and Azure AI Vision.

Before you start, follow these steps to set up your workspace:

Sign up for either an Azure free account or an Azure for Students account. If you already have an active subscription, you can use it.
Install Python 3.x, Visual Studio Code, Jupyter Notebook and Jupyter Extension for Visual Studio Code.
Create a Cognitive Services resource in the Azure portal.
Create an Azure Cosmos DB for PostgreSQL cluster and an Azure Blob Storage container.

What are vector embeddings?

In simple terms, vector embeddings are numerical representations of data, such as images, text, videos, and audio. These vectors are high-dimensional dense vectors, with each dimension containing information about the original content. By translating data into vectors, computers can capture the meaning of the data and understand the semantic similarity between two objects. We can quantify semantic similarity of two objects by their proximity in a vector space.

We can measure the semantic similarity by using a distance metric such as Euclidean distance, inner product, or cosine distance. In the following examples, we will use the cosine similarity, which is defined as the cosine of the angle between the two vectors.

There are numerous embedding models available, including OpenAI, Hugging Face, and Azure AI Vision. Azure AI Vision (formerly known as Azure Computer Vision) provides two Image Retrieval APIs for vectorizing image and text queries: the Vectorize Image API and the Vectorize Text API. This vectorization converts images and text into coordinates in a 1024-dimensional vector space, enabling users to search a collection of images using text and/or images without the need for metadata, such as image tags, labels, or captions.

How vector similarity search works?

Vector embeddings can be stored in a vector database, which is a specialized type of database optimized for storing and querying vectors with a large number of dimensions.

In the following video, you can learn more about vector embeddings and vector similarity search with Azure Cosmos DB for PostgreSQL and Azure AI Vision and walk through the Jupyter Notebooks that are available on my GitHub repository.

Python code samples

In my GitHub repository, you can find some Jupyter Notebooks to help you gain hands-on experience with the concepts introduced in the above video. You may want to refer to the README of the project for instructions on how to set up your Azure resources.

Quickstart

In the quickstart, you will explore the Image Retrieval APIs of Azure AI Vision and the basics of the pgvector extension. You will build a simple app to search a collection of images from a wide range of natural scenes. The images were taken from Kaggle.

Example scenario: Vector search with Azure Cosmos DB for PostgreSQL, Azure Blob Storage and Azure AI Vision

In this extended scenario, you will:

Use Azure Blob Storage to store approximately 8.2k images of paintings that were taken from the Paintings Dataset of the Visual Geometry Group - University of Oxford.
Generate the vector embedding of each image in the container using the Vectorize Image API of Azure AI Vision.
Store the vector embeddings along with a reference to the corresponding image file in a PostgreSQL table.
Execute SQL queries to search for images that are most similar to a reference image or a text prompt.

Resources

If you’d like to dive deeper into this topic, here are some helpful resources.

Automate customer reviews processing with Form Recognizer and Azure OpenAI

Foteini Savvidou — Mon, 10 Jul 2023 16:07:58 +0000

Organizations of all types rely on automated document processing and data extraction to streamline their operations. This article outlines a potential solution for automating PDF form processing, utilizing Azure Form Recognizer for data extraction and Azure OpenAI for "intelligent" data enrichment.

Data enrichment uses Artificial Intelligence (AI) to extract information, uncover patterns, and gain a deeper understanding of the data. This can be achieved through techniques such as key phrase extraction, named entity recognition, sentiment analysis, opinion mining, and custom models to identify essential information. To enrich the data, you can use the pretrained models of the Azure Cognitive Services for Language or train and deploy a custom model in Azure Machine Learning. In this post, I’ll demonstrate how to utilize the Davinci model of the Azure OpenAI Service to perform sentiment analysis and extract key phrases from customer service review forms.

In the two-part series "Automate document processing with Form Recognizer and Logic Apps" you learned how to train custom models in Azure Form Recognizer for extracting key-value pairs from documents and build an end-to-end form processing solution using Form Recognizer, Logic Apps, Azure Cosmos DB, and Power BI.

In this article, we will extend this pipeline by incorporating the Azure OpenAI service to enrich the extracted data. You will learn how to:

Create an Azure OpenAI resource and deploy a model.
Build a pipeline in Logic Apps to post a request to the Azure OpenAI endpoint, receive and evaluate the response.

Access to the Azure OpenAI service is currently limited. You can request access to the service by filling out the application form.

Architecture

The following architecture diagram illustrates the main components involved in the "intelligent" form processing solution that we are building and the information flow. In this post, we will focus on how to integrate the Azure OpenAI service into the pipeline that was built in the previous articles.

Dataflow

The information flow corresponding to the above architecture diagram is described as follows:

PDF forms or scanned images are (manually or programmatically) uploaded to a container in Azure Storage Account.
Whenever a form is uploaded to the specified container, an Event Grid event will trigger the Logic App to start processing the form.
The Logic App sends the URL of the file to the Form Recognizer API and receives the response data.
The Logic App posts a request to the Azure OpenAI service to analyze the sentiment and extract key phrases from the customer's review, and then receives the response data.
The extracted data are saved into a NoSQL database in Azure Cosmos DB.
Power BI is connected to Azure Cosmos DB to ingest the extracted data and provide dashboards.

Create an Azure OpenAI resource

You will create an Azure OpenAI resource through the Azure portal.

Sign in to the Azure portal and search for OpenAI.
In the Create Azure OpenAI page provide the following information:
- Subscription: Your Azure subscription.
- Resource group: Select an existing resource group or create a new one.
- Region: Choose any available region, for example, West Europe.
- Name: Enter a unique name.
- Pricing tier: Standard (S0).
Select Review + Create and then Create.
Once the deployment is complete, navigate to the Azure OpenAI Studio.

Deploy a model

Before you can use the Azure OpenAI service to generate text, you need to deploy a model. There are several available models in the Azure OpenAI Studio, each of which is tailored to a specific use case. In this post, you will deploy the text-davinci-003 model.

In the Azure OpenAI Studio, select Deployments under Management.
Click on + Create new deployment, and deploy a new model with the following settings:
- Model name: text-davinci-003
- Deployment name: Choose a memorable name for your deployment.
Once the model is deployed, you can test it in the Completions playground.
In the Completions playground, click View code. In the Sample Code window, save the endpoint of your deployed model. You will need this URL in the next step.

Build the workflow

In a previous article, you’ve built the workflow below. This pipeline receives a form that is uploaded in an Azure Storage container, extracts the fields from the form and saves the extracted data in Azure Cosmos DB.

You are going to extend this pipeline by adding an HTTP request action to send a request to the Azure OpenAI service. The workflow is illustrated in the following image.

You will build the form processing workflow using the Logic App Designer.

After the When a resource event occurs trigger, add two Initialize variable actions. Create two variables named openai-api-key and openai-url to store the Key and the URL of your Azure OpenAI resource, respectively.

The key of your Azure OpenAI service can be found by going to the Keys and Endpoint page of your resource in the Azure portal.

Under the Parse Fields block, select the plus (+) sign to add a new action. Select the HTTP action and enter the following information:

Method: POST
URI: openai-url
Headers: Content-Type: application/json, api-key: openai-api-key
Body:

    {
      "prompt": "You must extract the following information from the review below:\n1. Sentiment (key: Sentiment) (possible values: positive, negative, neutral, mixed)\n2. Opinion for the customer service in an array (key: KeyPhrases)\nAnswer the fields briefly and provide the results in JSON format using the keys above. For the second field, summarize the opinions if needed. If the review is empty, use \"NA\" for the first field and an empty array for the second field.\nReview: \"IF EXPRESSION\"",
      "temperature": 1,
      "top_p": 0.5,
      "frequency_penalty": 0,
      "presence_penalty": 0,
      "max_tokens": 100,
      "best_of": 1,
      "stop": null
    }

Replace IF EXPRESSION with the following expression, which extracts the value associated with the field Comments.

  if(contains(body('Parse_Fields')?['Comments'], 'content'), body('Parse_Fields')?['Comments']?['content'], '')

The output from the completions API will look as follows:

{
  "id": "ID of your call",
  "object": "text_completion",
  "created": 1680202256,
  "model": "text-davinci-003",
  "choices": [
    {
      "text": "text generated by the OpenAI API",
      "index": 0,
      "finish_reason": "stop",
      "logprobs": null
    }
  ],
  "usage": {
    "completion_tokens": 41,
    "prompt_tokens": 121,
    "total_tokens": 162
  }
}

Add a Parse JSON action to extract the response of your OpenAI model. Specify the following details:
- Content: Select Add dynamic content and find the block called Body.
- Schema: To generate the schema, use a sample JSON response of the OpenAI API.
Then, use a second Parse JSON action to extract the values generated by the OpenAI API for the keys named Sentiment and KeyPhrases. Enter the following information:
- Content: Select Add dynamic content and find the block called text under Parse OpenAI response.
- Schema: To generate the schema, use a sample JSON generated by the OpenAI API or the following schema:
```
{
    "properties": {
        "KeyPhrases": {
            "items": {
                "type": "string"
            },
            "type": "array"
        },
        "Sentiment": {
            "type": "string"
        }
    },
    "type": "object"
}
```
The Logic App automatically adds a For each block around the Parse JSON block.
Modify the existing Compose action to include the information generated by Azure OpenAI. To extract the values associated with the fields Sentiment and KeyPhrases, select Add dynamic content and choose the respective blocks.
Modify the Create or update document (V3) action as needed.

Save the workflow and then click Run Trigger > Run. Upload a file to your Azure Storage container to test the Logic App.

Visualize the data in Power BI

You can use Power BI to visualize the results obtained from the form processing workflow. If you don’t have a Power BI subscription, you can use Power BI Desktop, which is a free service.

Open Power BI Desktop and in the Home tab select Get data > More…
Choose the Azure Cosmos DB v1 connection and click Connect.
In the pop-up window, enter the URL of your Cosmos DB account and the id of your database and collection. Then, click OK.
Once Power BI is connected to your Cosmos DB account, you can see the stored data and transform it.

Below you can see a simple Power BI dashboard that I created to visualize the data generated by the Azure OpenAI service.

Summary and next steps

In this article, you created an end-to-end automated form processing solution using Form Recognizer, Logic Apps, and Azure OpenAI. You can use this solution to create automated workflows for your specific scenarios. You can also extend this scenario by integrating Azure Cognitive Services for Language into your Logic App workflow.

You can learn more about Azure OpenAI in the resources below:

👋 Hi, I am Foteini Savvidou!
An Electrical and Computer Engineering student and Microsoft AI MVP (Most Valuable Professional) from Greece.

🌈 LinkedIn | Blog | GitHub

Automate document processing with Form Recognizer and Logic Apps (Part 2)

Foteini Savvidou — Mon, 10 Jul 2023 13:51:50 +0000

Processing of forms and documents is part of several scenarios both in business and in everyday life. Manual data extraction from documents, either in electronic or printed format, is time-consuming, costly, and error-prone.

In the first part, you trained two custom models for extracting key-value pairs from customer service review forms and composed these models together into a single model. In this article, you will build an automated form processing solution.

You will learn how to:

Create a Logic App workflow that responds to Event Grid events.
Integrate Form Recognizer into a Logic Apps workflow.
Store the extracted information from the Form Recognizer’s model in Azure Cosmos DB.
Use Power BI to visualize the insights from the analysis of the forms.

To build this solution, you will need an Azure subscription. If you don’t have one, you can sign up for an Azure free account.

Scenario details

Consider that you work in a company that provides customer service to a variety of customers. Every day, customers leave reviews about their experiences with the customer service they received. You need to analyze these reviews in order to identify areas of improvement and to track customer satisfaction. These reviews are submitted as paper forms or PDFs and may include up to two different form types. To make this process easier, we need to use a single service to analyze all the reviews.

In the following image, you can see the workflow that we will build to automate the process of extracting and analyzing customer service reviews. In this post, we will focus on the second and third steps of the workflow.

Architecture

The following architecture diagram illustrates the main components involved in the automated form processing solution that we are building and the information flow. The system receives the forms (either in PDF or scanned image format), extracts the fields from the form and saves the extracted data in Azure Cosmos DB. Power BI is then used to visualize the insights from the data.

Dataflow

The information flow corresponding to the above architecture diagram is described as follows:

PDF forms or scanned images are (manually or programmatically) uploaded to a container in Azure Storage Account.
Whenever a form is uploaded to the specified container, an Event Grid event will trigger the Logic App to start processing the form.
The Logic App sends the URL of the file to the Form Recognizer API and receives the response data.
The extracted key-value pairs are saved into a NoSQL database in Azure Cosmos DB.
Power BI is connected to Azure Cosmos DB to ingest the extracted data and provide dashboards.

Components

Setup Azure Blob Storage

You will create a storage account and a container to upload the forms that will be processed by our Logic App.

Sign in to the Azure portal, search for Storage accounts and then select Create.
Create a storage account with the following settings:
- Subscription: Your Azure subscription.
- Resource group: Select an existing resource group (i.e. the resource group that you created in the Part 1) or create a new one.
- Storage account name: Enter a unique name.
- Region: Choose any available region, for example, West Europe.
- Performance: Standard.
- Redundancy: Locally-redundant storage (LRS).
Select Review + Create and then select the Create button and wait for the deployment to complete. Once the deployment is complete, navigate to your storage account.
In the left pane, under Data storage, select Containers.
Create a new container. Select a name and set the public access level to Container.

Setup Event Grid

Event Grid is a kind of event broker that you can use to integrate applications using events. We will use Event Grid as a trigger to run the Logic App when a file is uploaded in the Azure Storage container.

To use Event Grid, you must first register it as a resource provider in your subscription.

Setup Azure Cosmos DB

You need to create an Azure Cosmos DB account, database, and container to store the fields extracted from the safety forms.

In the Azure portal, search for Azure Cosmos DB and then click Create.
Create a new Azure Cosmos DB for NoSQL account by selecting the corresponding card in the Which API best suits your workload? window.
Create an Azure Cosmos DB account with the settings below:
- Subscription: Your Azure subscription.
- Resource group: Select an existing resource group (i.e. the resource group that you created in the Part 1) or create a new one.
- Account name: Enter a unique name.
- Location: Choose any available region, for example, West Europe.
- Capacity mode: Provisioned throughput.
- Limit total account throughput: selected.
Select Review + Create and then select the Create button and wait for the deployment to complete. Once the deployment is complete, navigate to your resource.
In the left pane, select Data Explorer. Then, create a new database and a container.

Create a Logic App

The last component that we need to provision is a Logic App.

In the Azure portal, search for Logic apps and then click + Add.
Create a Logic App by specifying the following parameters:
- Subscription: Your Azure subscription.
- Resource group: Select an existing resource group (i.e. the resource group that you created in the Part 1) or create a new one.
- Logic App name: Enter a unique name.
- Region: Choose any available region, for example, West Europe.
- Plan type: Consumption.
- Zone redundancy: Disabled.
Select Review + Create and then Create.
Once the deployment is complete, navigate to your Logic App resource. You are now ready to build the document processing workflow.

Build the workflow

You will build the document processing workflow using the Logic App Designer, a graphical user interface that enables you to create workflows visually.

Once the Logic App resource is created, you will see the starter window of Logic App designer. In the Start with a common trigger window, select the When an Event Grid event occurs block.

If you can’t find this block, create an empty workflow and then select the When a resource event occurs trigger.
Sign in to your account. In the When a resource event occurs block specify the following details:
- Subscription: Select your subscription.
- Resource Type: Microsoft.Storage.StorageAccounts
- Resource Name: The name of your Storage account.
- Event Type Item – 1: Microsoft.Storage.BlobCreated
In the Event Grid trigger block, you can optionally apply filters. For example, you can use the Prefix Filter to subscribe to events from a specific container in the Storage account or use the Suffix Filter to filter events based on the extension of the uploaded file, such as .jpg or .pdf.
Add a new step. In the Choose an operation window, search for Parse JSON and then select the respective action.

In the Parse JSON block, enter the following parameters:

Content: Body
Schema: Use the following schema.

   {
       "properties": {
           "data": {
               "properties": {
                   "api": {
                       "type": "string"
                   },
                   "blobType": {
                       "type": "string"
                   },
                   "clientRequestId": {
                       "type": "string"
                   },
                   "contentLength": {
                       "type": "integer"
                   },
                   "contentType": {
                       "type": "string"
                   },
                   "eTag": {
                       "type": "string"
                   },
                   "requestId": {
                       "type": "string"
                   },
                   "sequencer": {
                       "type": "string"
                   },
                   "storageDiagnostics": {
                       "properties": {
                           "batchId": {
                               "type": "string"
                           }
                       },
                       "type": "object"
                   },
                   "url": {
                       "type": "string"
                   }
               },
               "type": "object"
           },
           "dataVersion": {
               "type": "string"
           },
           "eventTime": {
               "type": "string"
           },
           "eventType": {
               "type": "string"
           },
           "id": {
               "type": "string"
           },
           "metadataVersion": {
               "type": "string"
           },
           "subject": {
               "type": "string"
           },
           "topic": {
               "type": "string"
           }
       },
       "type": "object"
   }

Find the Analyze Document for Prebuilt or Custom models (v3.0 API) in the Form Recognizer connector and add it to your workflow.
Connect to Form Recognizer:
- Connection name: Enter a name for the connection.
- Endpoint URL: Enter the Endpoint URL of your Form Recognizer resource.
- Account Key: Enter the Key 1 of your Form Recognizer resource.
You can find the Endpoint URL and the Key in your Form Recognizer resource, under Keys and Endpoint.
Then, specify the following parameters:
- Model Identifier: Enter the name of the model that will be used to analyze the forms (i.e. the name of the composed model that you created in Part 1).
- Document/Image URL: Click Add dynamic content and select the url.

Save the extracted data

The JSON response of the Analyze Document for Prebuilt or Custom models (v3.0 API) action includes several information, such as the text extracted from the form, layout information, selection marks, and key-value pairs. We are interested in extracting the fields and their associated values. In the following steps, you will build a pipeline that extracts the fields from the JSON response, processes the values, generates a JSON file and saves the file in Azure Cosmos DB.

Add a Parse JSON action and specify the following details:
- Content: Select Add dynamic content and find the block called fields.
- Schema: To generate the schema, use a sample JSON response of the Form Recognizer action. Click Use sample payload to generate schema and paste the section of the Form Recognizer’s response identified by the term "fields".
The Logic App automatically adds a For each block around the Parse JSON block.

Add a Compose action (inside the For each block). We will save the extracted fields in the following format:

{
  "BillingPayment": "value1",
  "Comments": "value2",
  "OtherService": "value3",
  "ProductServiceInformation": "value4",
  "Recommend": "value5",
  "Satisfaction": "value6",
  "Suggestions": "value7",
  "TechnicalSupport": "value8",
  "id": "id"
}

To extract the value associated with each field, use if statements. You can add an if statement by selecting Add dynamic content > Expression.

For example, to extract the values associated with the fields BillingPayment, Comments and Recommend, you can use the below expressions:

// BillingPayment
if(and(contains(body('Parse_Fields')?['BillingPayment'], 'content'), equals(body('Parse_Fields')?['BillingPayment']?['valueSelectionMark'], 'selected')), 1, 0)

// Comments
if(contains(body('Parse_Fields')?['Comments'], 'content'), body('Parse_Fields')?['Comments']?['content'], 'NA')

// Recommend
if(and(contains(body('Parse_Fields')?['RecommendVeryLikely'], 'content'), equals(body('Parse_Fields')?['RecommendVeryLikely']?['valueSelectionMark'], 'selected')), 'Very likely', if(and(contains(body('Parse_Fields')?['RecommendLikely'], 'content'), equals(body('Parse_Fields')?['RecommendLikely']?['valueSelectionMark'], 'selected')), 'Likely', if(and(contains(body('Parse_Fields')?['RecommendNeutral'], 'content'), equals(body('Parse_Fields')?['RecommendNeutral']?['valueSelectionMark'], 'selected')), 'Neutral', if(and(contains(body('Parse_Fields')?['RecommendUnlikely'], 'content'), equals(body('Parse_Fields')?['RecommendUnlikely']?['valueSelectionMark'], 'selected')), 'Unlikely', if(and(contains(body('Parse_Fields')?['RecommendVeryUnlikely'], 'content'), equals(body('Parse_Fields')?['RecommendVeryUnlikely']?['valueSelectionMark'], 'selected')), 'Very unlikely', 'NA')))))

In a similar way, you can extract the values corresponding to the remaining fields.

Click Add an action and select the Create or update document (V3) action of the Azure Cosmos DB connector. Then, configure the Azure Cosmos DB connection by specifying the values below:
- Connection name: Enter a name for the connection.
- Authentication type: Access Key.
- Access Key to your Cosmos DB account: Enter the primary (or secondary) key of your Azure Cosmos DB resource.
- Account ID: Enter the name of your Azure Cosmos DB account.
To find the primary key of your Cosmos DB account, navigate to your Cosmos DB resource and select Keys.
In the Create or update document (V3) block, specify the ID of your database and collection. Then, under Document, select Add dynamic content and add the block named Outputs.

Save the workflow and then click Run Trigger > Run. Upload a file to your Azure Storage container to test the Logic App.

Connect Power BI to Azure Cosmos DB

You can use Power BI to visualize the results obtained from the form processing workflow. If you don’t have a Power BI subscription, you can use Power BI Desktop, which is a free service.

Open Power BI Desktop and in the Home tab select Get data > More…
Choose the Azure Cosmos DB v1 connection and click Connect.
In the pop-up window, enter the URL of your Cosmos DB account and the id of your database and collection. Then, click OK.
Once Power BI is connected to your Cosmos DB account, you can see the stored data and transform it.

Below you can see a simple Power BI dashboard that I created to visualize the extracted data.

Summary and next steps

In this article, you created an end-to-end automated form processing solution using Form Recognizer, Logic Apps, Azure Cosmos DB and Power BI. You can use this solution to create automated workflows for your specific scenarios. You can also extend this scenario by adding AI capabilities (such as, sentiment analysis, key-phrase extraction, opinion mining) in your Logic App workflow.

Here are some additional scenarios that leverage Azure Form Recognizer:

Clean-up

If you have finished learning, you can delete the resource group from your Azure subscription:

In the Azure portal, select Resource groups on the right menu and then select the resource group that you have created.
Click Delete resource group.

👋 Hi, I am Foteini Savvidou!
An Electrical and Computer Engineering student and Microsoft AI MVP (Most Valuable Professional) from Greece.

🌈 LinkedIn | Blog | GitHub

Automate document processing with Form Recognizer and Logic Apps (Part 1)

Foteini Savvidou — Sun, 09 Jul 2023 10:51:19 +0000

Azure Form Recognizer is an Applied AI Service that enables you to extract text, table data, key-value pairs, and layout information from forms and documents. In this two-part series, you will learn how to build an end-to-end document processing automation solution utilizing Azure Form Recognizer, Logic Apps, Azure Cosmos DB, and Power BI. In the first part, you will be introduced to the Form Recognizer's features and train a custom model for extracting key-value pairs from customer service review forms.

You will learn how to:

Provision a Form Recognizer resource.
Train a custom model in Form Recognizer Studio.
Compose two custom template models together into a single model.

To build this solution, you will need an Azure subscription. If you don’t have one, you can sign up for an Azure free account.

What is Azure Form Recognizer?

Azure Form Recognizer is a cloud-based service that applies machine learning-based Optical Character Recognition (OCR) and document understanding technologies to extract data from forms and documents. It provides numerous pre-built models for analyzing common documents, such as invoices, receipts, and business cards, while you can also build custom models to analyze documents specific to your business.

Study the following sketch note to learn more about the Form Recognizer’s features.

Scenario details

In the following image, you can see the workflow that we will build to automate the process of extracting and analyzing customer service reviews. In this post, we will focus on the first step of the workflow: Train a model in Azure Form Recognizer.

Create a Form Recognizer resource

To use Form Recognizer, you need to create a Cognitive Services or a Form Recognizer resource. In this article, I’ll show you how to create a single-service resource in the Azure portal.

Cognitive Services resource is a multi-service resource that provides access to the entire collection of Azure Cognitive Services under a single endpoint and key.

Sign-in to the Azure portal and select Create a resource.
Search for Form Recognizer and then click Create.
Create a Form Recognizer resource with the following settings:
- Subscription: Your Azure subscription.
- Resource group: Select an existing resource group or create a new one.
- Region: Choose any available region, for example, West Europe.
- Name: This would be your custom domain name in your endpoint. Enter a unique name.
- Pricing tier: Free F0.
Select Review + Create and then select the Create button and wait for the deployment to complete.
Navigate to the Form Recognizer Studio and sign in.

If this is your first time using Form Recognizer Studio, configure your service resource by selecting your subscription, resource group, and the existing Form Recognizer resource.

Additional resources for custom projects

To train a custom model, in addition to the Form Recognizer resource, you need to create an Azure Storage Account and a container to upload your training dataset.

In the Azure Portal, search for Storage accounts and then select Create.
Create a storage account with the following settings:
- Subscription: Your Azure subscription.
- Resource group: Select the resource group that you created in the previous step.
- Storage account name: Enter a unique name.
- Region: Choose any available region, for example, West Europe.
- Performance: Standard.
- Redundancy: Locally-redundant storage (LRS).
Select Review + Create and then select the Create button and wait for the deployment to complete. Once the deployment is complete, navigate to your storage account.
In the left pane, under Data storage, select Containers.
Create a new container. Select a name and set the public access level to Container.
In the left pane, under Settings, select Resource sharing (CORS). Create a new CORS entry and set the Allowed origins to https://formrecognizer.appliedai.azure.com.

Now you can upload your sample documents set to the container that you created.

You can organize your data in folders and then specify the folder path in the custom project creation window in Form Recognizer Studio.

Create a custom project

In the Form Recognizer Studio, select the Custom extraction models card.
Create a new project and specify the following details:
- Project name: Customer Service Reviews - Set 1
- Configure service resource: Select your subscription, resource group, and Form Recognizer resource. In the API version, you can choose the 2022-08-31 (General Availability) API.
- Connect training data source: Select your subscription, resource group, storage account, and blob container. If you’ve organized your data in folders, specify the folder path in the corresponding field.
Select Create project.

Label data and train model

In the Label data view, define the labels that you want to extract from your documents and their types (field, selection mark, signature, or table). For labels of type Field, you can additionally specify their sub-type, such as string or number.
Select a document from your dataset. Form Recognizer will run the document through the Layout API to extract text and layout information.
Then, you can start labeling your documents by selecting the text or selection mark in the document and choosing the appropriate label from the drop-down list.

When labeling selection marks, select only the selection mark, without the surrounding text.
Once you have labeled at least five documents, select the Train option.
In the Train a new model window, ender a Model ID and set the Build Mode to Template.

Compose models

Once you’ve trained a template model for each document layout variation, you can compose them together into a single model. Then, you can use the composed model in your applications.

Summary and next steps

In this article, you learned how to train a custom template model in Azure Form Recognizer and create a composed model. In the next post, you will use the composed model in an Azure Logic Apps workflow to build an automated form processing solution.

If you want to improve the accuracy of the template model that you’ve trained, you can check out the following resources:

Clean-up

If you have finished learning, you can delete the resource group from your Azure subscription:

In the Azure portal, select Resource groups on the right menu and then select the resource group that you have created.
Click Delete resource group.

👋 Hi, I am Foteini Savvidou!
An Electrical and Computer Engineering student and Microsoft AI MVP (Most Valuable Professional) from Greece.

🌈 LinkedIn | Blog | GitHub