Forem: Neel

Opensearch as a Vector Database

Neel — Fri, 05 Dec 2025 08:12:43 +0000

LLM, RAG, Embeddings, Vector Database. These were all buzz words to get everyone's attention a couple of years ago, but not anymore. Using an effective vector Database to perform Retrieval Augmented Generation or Semantic Search is mainstream now and with opensearch, you can make the process simpler

🌟 No Extra Step for Embeddings

Opensearch ingestion pipelines make it easier to generate vector embeddings for the data in flight with no hassle. This means two things,

You don't need to write custom logic to convert the text content to a vector type and store it
You don't need to write custom logic to convert the search query to a vector type

The ingestion pipeline makes use of the builtin sentence transformer models provided by Opensearch to generate the embeddings for the text content. Opensearch comes bundled with a collection of pre-trained models from huggingface and the model that best suites your purpose can be selected to generate the embeddings

🛠️ The Setup

🔧 Prerequisite

By default, Opensearch will run the models on a dedicated ML node. If you do not have a dedicated ML node (or if you are using the AWS managed Opensearch service), then this restriction can be disabled using the following cluster settings.

PUT _cluster/settings
{
  "persistent": {
    "plugins.ml_commons.only_run_on_ml_node": "false",
    "plugins.ml_commons.model_access_control_enabled": "true",
    "plugins.ml_commons.native_memory_threshold": "99"
  }
}

Step - 1: Model deployment 🤖

Lets start with creating a new model group first. Use the Opensearch dashboard to execute the following request to create a custom model group

POST /_plugins/_ml/model_groups/_register
{
  "name": "optimus_embedding_group",
  "description": "A custom and powerful model group"
}

On completion of this request, a model group ID will be returned. Keep that group ID safely for the model deployment

The next part is to create an instance of the pre-trained model that you can deploy. This step requires you to choose the right model for your use case. As we are covering semantic search in this article, I have chosen the msmarco-distilbert-base-tas-b which generates a 768 dimension vector space which is optimal for semantic search

POST /_plugins/_ml/models/_register
{
  "name": "huggingface/sentence-transformers/msmarco-distilbert-base-tas-b",
  "version": "1.0.2",
  "model_group_id": "<model group ID obtained from the previous step>",
  "model_format": "TORCH_SCRIPT"
}

This request is asynchronous and on completion, it returns a task ID for the registration task that is running in the background. Obtain the task ID and check its status periodically using the following request

GET /_plugins/_ml/tasks/<task ID from the previous request>

The response should have the state as COMPLETED and the field model_id should be present in the response. Once the model_id becomes available, use that to deploy the model. Ensure that you keep note of this model ID as its required for all future purposes

POST /_plugins/_ml/models/<model ID fetched once the task completes>/_deploy

This makes the model available to generate the text embeddings on the go

Step - 2: Ingestion Pipeline Setup 🚚

The ingestion pipeline acts as a trigger for an index and performs the required transformation in realtime. The pipelines have different types of ingest processors and we will be making use of the Text/image embedding processor to generate the text embeddings for the searchable content

Execute the following request to setup a new ingestion pipeline

PUT /_ingest/pipeline/<a_name_for_the_pipeline>
{
    "description": "Optimus pipeline, a super powerful pipeline for generating embeddings",
    "processors": [
        {
            "text_image_embedding": {
                "model_id": "<The model_id from the previous step>",
                "embedding": "text_embeddings",
                "field_map": {
                    "text": "article"
                }
            }
        }
    ]
}

Explanation:

The embedding field instructs the pipeline to store the vector data to the mentioned field - text_embeddings
The field_map field defines the searchable field that needs to be embedded

In our sample, we are going to generate embeddings for a field called article and the pipeline is supposed to store the embeddings to another field called text_embeddings

Step - 3: Index Creation 🗂️

We will keep the index simple and, the focus is on the vector field and the ingestion pipeline

Use the following request to create a new index

PUT /news_articles
{
  "settings": {
    "index.knn": true,
    "default_pipeline": "<name of the pipeline used during creation>"
  },
  "mappings": {
    "properties": {
      "article": {
        "type": "text"
      },
      "text_embeddings": {
        "type": "knn_vector",
        "dimension": 768,
        "method": {
          "engine": "lucene",
          "space_type": "l2",
          "name": "hnsw",
          "parameters": {}
        }
      }
    }
  }
}

In the index settings, we have enabled KNN which is required for the semantic search and configured the default_pipeline. Whenever a insert or update operation happens on the index, the ingestion pipeline will be invoked to generate the text embeddings for the article field

Step - 4: The Search! 🔍

Lets load the index with some sample data and see how to perform the search operation

The following are some sample news articles and we will be performing search on these

POST /news_articles/_doc/3
{
  "article": "..."
}

To search using vector similarity, Opensearch provides the neural search feature. This uses cosine similarity to rank the top k closest items

Use the following request to perform the vector search

GET /news_articles/_search
{
  "query": {
    "bool": {
      "should": [
        {
          "neural": {
            "text_embeddings": {
              "query_text": "climate crisis",
              "model_id": "<ID of the deployed model>",
              "k": 1
            }
          }
        }
      ]
    }
  },
  "_source": {
    "excludes": "text_embeddings"
  }
}

Explanation:

In the above search query, I am explicitly asking for the top k=1 similar items because there are only 3 documents in the sample dataset. The value of k can be increased to get how many ever number of similar documents you want and it will be ranked in the descending order of similarity (or the ascending order of vector distance)

The above sample search term "climate crisis" will rank the document with the "news about the floods" as the top most result

If I search for "developments in artificial intelligence", the "article about OpenAI" will be ranked on top

If I search for "team that ruined the GOAT", the "article about Ferrari" will be ranked on top (not joking btw 😜)

Few add-ons

For a realistic use case, the document could have more than one field and you may want to perform the search combining the content from multiple fields. In such cases, you can use the script ingest processor within the pipeline to normalize the content and use the normalized field to generate the embeddings. A sample can be found here
If you wish to combine the similarity search along with a normal phrase based search or something else, then it can be achieved by combining multiple queries within a boolean query and using query boosting to give preference to the required query criteria

GET /news_articles/_search
{
  "query": {
    "bool": {
      "should": [
        {
          "neural": {
            "text_embeddings": {
              "query_text": "race",
              "model_id": "<ID of the model>",
              "k": 5,
              "boost": 1.5
            }
          }
        },
        {
          "bool": {
            "must": [
              {
                "query_string": {
                  "query": "race",
                  "default_field": "all",
                  "default_operator": "AND",
                  "analyze_wildcard": true,
                  "boost": 1.7
                }
              }
            ]
          }
        }
      ]
    }
  },
  "_source": {
    "excludes": "text_embeddings"
  }
}

🖋️Conclusion

If Opensearch is your go to data source for search then the combination of the builtin models along with the ingestion pipeline will make it a powerhouse for semantic search or similarity search

🔍🔍 Happy Coding!!! 🔍🔍

DynamoDB with Enhanced Client - Schemas and data serialization made easy

Neel — Sun, 08 Jun 2025 12:52:22 +0000

DynamoDB is a powerful NoSQL Database offered and managed by AWS. Flexible schema is a key feature offered by NoSQL systems and DynamoDB is no exception. Even though flexible schemas may sound appealing, it is better to keep the schema in control to ensure the tables don't go out of control with different documents having different fields with no correlation

The AWS SDK-v2 for Java offers a high level API called the EnhancedClient to setup a strict table schema for the documents and ensure consistent serialization and de-serialization of data across the application. Let's see this in action

I assume that the readers already know their way around Spring Boot and DynamoDB, so I am skipping the bare minimum parts like table creation and stuffs

Blueprint

I will be using DynamoDB Local for this demo. Ensure that you run the local JAR with sharedDb mode, otherwise the local DB will create different accounts for different clients.

The app will have the following endpoints to perform the Create and Read operations

POST /api/v1/book
GET /api/v1/book/id
GET /api/v1/books
GET /api/v1/books/year/{year_published}

The following is going to be the structure of a document to be stored in the Books table

id - Partition Key
We will add a new GSI with the yearPublished as the Partition Key

{
  "id": 1,
  "title": "Chip War",
  "description": "The Fight for the World's Most Critical Technology",
  "isbn": 9781982172008,
  "author": {
    "name": "Chris Miller",
    "biography": "Historian and book author",
    "otherBooksPublished": [
      {
        "isbn": 9780316057011,
        "title": "The Real Animal House"
      }
    ]
  },
  "yearPublished": 2022,
  "languagesAvailableIn": ["English", "Spanish", "German"],
  "currentEdition": 1,
  "averageRating": 4.44,
  "genere": ["science", "politics"],
  "createdAt": 1749283110
}

Notice that I have added nested fields and different types of data specifically to show the schema mapping using the EnhancedClient API

Lets Cook!

DynamoDB is running as a docker container on my machine and I have already bootstrapped a new springboot project with just spring-web and lombok as the dependencies.

Client Setup

To setup the EnhancedClient API, the following dependencies should be added to the project

software.amazon.awssdk:dynamodb:2.31.58
software.amazon.awssdk:dynamodb-enhanced:2.31.58

The following code shows the basic setup for a singleton Enhanced Client pointing to the local DynamoDB instance running on port 8000. To connect to the actual DynamoDB instance on AWS, the endpointOverride should be removed and the credentials should be configured based on your application setup. Refer this for more information

package com.itassistors.books;

import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import software.amazon.awssdk.enhanced.dynamodb.DynamoDbEnhancedClient;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.dynamodb.DynamoDbClient;

import java.net.URI;

@Configuration
public class DynamoConfig {
    @Bean
    DynamoDbEnhancedClient dynamoDbEnhancedClient() {
        DynamoDbClient dbClient = DynamoDbClient.builder()
                .region(Region.US_EAST_1)
                .endpointOverride(URI.create("http://localhost:8000")) // Use the local DynamoDB endpoint
                .build();

        return DynamoDbEnhancedClient.builder()
                .dynamoDbClient(dbClient)
                .build();
    }
}

Bean and TableSchema<T> setup

The beauty of using the EnhancedClient API is the control you get over the schema of the table and robust de-/serialization of the data stored in DynamoDB. We will create the required classes in Java to handle the data similar to the sample JSON data mentioned above, starting with the Book.java class, Author.java and OtherBooksPublished.java for the nested data

package com.itassistors.books.entity;

import lombok.AllArgsConstructor;
import lombok.Data;
import software.amazon.awssdk.enhanced.dynamodb.EnhancedType;
import software.amazon.awssdk.enhanced.dynamodb.TableSchema;
import software.amazon.awssdk.enhanced.dynamodb.mapper.StaticAttributeTags;
import software.amazon.awssdk.enhanced.dynamodb.mapper.annotations.DynamoDbBean;

import java.util.List;

@AllArgsConstructor
@Data
@DynamoDbBean
public class Book {
    private Long id;
    private String title;
    private String description;
    private Long isbn;
    private Author author;
    private Integer yearPublished;
    private List<String> languagesAvailableIn;
    private Integer currentEdition;
    private Double averageRating;
    private List<String> genres;
    private Long createdAt;

    public Book() {
    }

    public static final TableSchema<Book> TABLE_SCHEMA = TableSchema.builder(Book.class)
            .newItemSupplier(Book::new)
            .addAttribute(Long.class, a -> a.name("id")
                    .getter(Book::getId)
                    .setter(Book::setId)
                    .tags(StaticAttributeTags.primaryPartitionKey()))
            .addAttribute(String.class, a -> a.name("title")
                    .getter(Book::getTitle)
                    .setter(Book::setTitle))
            .addAttribute(String.class, a -> a.name("description")
                    .getter(Book::getDescription)
                    .setter(Book::setDescription))
            .addAttribute(Long.class, a -> a.name("isbn")
                    .getter(Book::getIsbn)
                    .setter(Book::setIsbn))
            .addAttribute(EnhancedType.documentOf(Author.class, Author.TABLE_SCHEMA), a -> a.name("author")
                    .getter(Book::getAuthor)
                    .setter(Book::setAuthor))
            .addAttribute(Integer.class, a -> a.name("yearPublished")
                    .getter(Book::getYearPublished)
                    .setter(Book::setYearPublished)
                    .addTag(StaticAttributeTags.secondaryPartitionKey("yearPublishedIndex")))
            .addAttribute(EnhancedType.listOf(String.class), a -> a.name("languagesAvailableIn")
                    .getter(Book::getLanguagesAvailableIn)
                    .setter(Book::setLanguagesAvailableIn))
            .addAttribute(Integer.class, a -> a.name("currentEdition")
                    .getter(Book::getCurrentEdition)
                    .setter(Book::setCurrentEdition))
            .addAttribute(Double.class, a -> a.name("averageRating")
                    .getter(Book::getAverageRating)
                    .setter(Book::setAverageRating))
            .addAttribute(EnhancedType.listOf(String.class), a -> a.name("genres")
                    .getter(Book::getGenres)
                    .setter(Book::setGenres))
            .addAttribute(Long.class, a -> a.name("createdAt")
                    .getter(Book::getCreatedAt)
                    .setter(Book::setCreatedAt))
            .build();
}

package com.itassistors.books.entity;

import lombok.Data;
import software.amazon.awssdk.enhanced.dynamodb.EnhancedType;
import software.amazon.awssdk.enhanced.dynamodb.TableSchema;

import java.util.List;

@Data
public class Author {
    private String name;
    private String biography;
    private List<OtherBooksPublished> otherBooksPublished;

    public static final TableSchema<Author> TABLE_SCHEMA = TableSchema.builder(Author.class)
            .newItemSupplier(Author::new)
            .addAttribute(String.class, a -> a.name("name")
                    .getter(Author::getName)
                    .setter(Author::setName))
            .addAttribute(String.class, a -> a.name("biography")
                    .getter(Author::getBiography)
                    .setter(Author::setBiography))
            .addAttribute(EnhancedType.listOf(EnhancedType.documentOf(OtherBooksPublished.class, OtherBooksPublished.TABLE_SCHEMA)),
                    a -> a.name("otherBooksPublished")
                            .getter(Author::getOtherBooksPublished)
                            .setter(Author::setOtherBooksPublished))
            .build();
}

package com.itassistors.books.entity;

import lombok.Data;
import software.amazon.awssdk.enhanced.dynamodb.TableSchema;

@Data
public class OtherBooksPublished {
    private Long isbn;
    private String title;

    public static final TableSchema<OtherBooksPublished> TABLE_SCHEMA = TableSchema.builder(OtherBooksPublished.class)
            .newItemSupplier(OtherBooksPublished::new)
            .addAttribute(Long.class, a -> a.name("isbn")
                    .getter(OtherBooksPublished::getIsbn)
                    .setter(OtherBooksPublished::setIsbn))
            .addAttribute(String.class, a -> a.name("title")
                    .getter(OtherBooksPublished::getTitle)
                    .setter(OtherBooksPublished::setTitle))
            .build();
}

I have added the TableSchema built for all the three classes as constant static variables for ease of use and to ensure those are immutable

A few key things to note here are…

The newItemSupplier attribute is required to create new instances of the class for de-seralizing the JSON data fetched from DynamoDB
For all the required attributes stored in the table, the appropriate getters and setters should be setup using the addAttribute method. If a particular attribute of the class is not stored to the DB, then such fields could be ignored from the TableSchema
Key fields must be attributed using the addTag method to let the client know on how to handle those fields. I have added the tags for the primary partitionKey - id, the GSI partitionKey - yearPublished. For sort keys, the same method can be used with the primarySortKey method
If the data stored in your table is going to be flat without any nesting, then you need not go through the process of creating explicit table schemas. It can be achieved easily using TableSchema.fromBean(Bean.class), but that’s not the case here
- To handle nested or complex types such as lists and objects, you can make use of the EnhancedType.documentOf and EnhancedType.listOf wherever required. You need to ensure that you handle the enhanced types properly to avoid any de-serialization errors

Service Layer

Lets create a service class to power the APIs. We will start with injecting the DynamoDB enhanced client and the methods for working with the DB

package com.itassistors.books;

import com.itassistors.books.entity.Book;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Service;
import software.amazon.awssdk.core.pagination.sync.SdkIterable;
import software.amazon.awssdk.enhanced.dynamodb.DynamoDbEnhancedClient;
import software.amazon.awssdk.enhanced.dynamodb.DynamoDbIndex;
import software.amazon.awssdk.enhanced.dynamodb.DynamoDbTable;
import software.amazon.awssdk.enhanced.dynamodb.Key;
import software.amazon.awssdk.enhanced.dynamodb.model.Page;
import software.amazon.awssdk.enhanced.dynamodb.model.PageIterable;
import software.amazon.awssdk.enhanced.dynamodb.model.QueryConditional;
import software.amazon.awssdk.enhanced.dynamodb.model.QueryEnhancedRequest;

import java.time.Instant;
import java.util.List;

@Slf4j
@Service
public class BookService {
    private final DynamoDbTable<Book> booksTable;

    public BookService(@Value("${db.table.books:books}") String booksTableName,
                       DynamoDbEnhancedClient dynamoDbEnhancedClient) {
        this.booksTable = dynamoDbEnhancedClient.table(booksTableName, Book.TABLE_SCHEMA);
    }

    public List<Book> getAllBooks() {
        PageIterable<Book> results = booksTable.scan();

        log.info("Fetched {} books from the database", results.stream().count());
        return results.items().stream().toList();
    }

    public Book addBook(Book book) {
        book.setId(Instant.now().getEpochSecond());
        book.setCreatedAt(System.currentTimeMillis());

        booksTable.putItem(book);
        log.info("Added book: {}", book.getTitle());

        return book;
    }

    public Book getBookById(Long id) {
        log.info("Fetching book by ID: {}", id);
        Key key = Key.builder().partitionValue(id).build();
        return booksTable.getItem(key);
    }

    public List<Book> getBookByYearPublished(Integer yearPublished) {
        DynamoDbIndex<Book> isbnIndex = booksTable.index("yearPublishedIndex");
        log.info("Querying book by year published: {}", yearPublished);

        QueryConditional queryConditional = QueryConditional.keyEqualTo(Key.builder()
                .partitionValue(yearPublished)
                .build());
        QueryEnhancedRequest queryEnhancedRequest = QueryEnhancedRequest.builder()
                .queryConditional(queryConditional)
                .build();

        SdkIterable<Page<Book>> results = isbnIndex.query(queryEnhancedRequest);
        log.info("Found {} books published in the year {}", results.stream().count(), yearPublished);
        if (results.stream().findAny().isEmpty()) {
            log.warn("No books found for the year {}", yearPublished);
            return List.of();
        }

        List<Book> books = results.stream()
                .flatMap(page -> page.items().stream())
                .toList();

        log.info("Returning {} books published in the year {}", books.size(), yearPublished);
        return books;
    }
}

In the above code, I have created a single instance for the DynamoDB table

this.booksTable = dynamoDbEnhancedClient.table(booksTableName, Book.TABLE_SCHEMA);

The table instances are supposed to be immutable and are eligible singleton candidates. For this simple application, this setup is sufficient, but for large scale applications you need to create a singleton for instantiating the table objects that can be used across the entire application without re-creating it

The rest of the code should be pretty self explanatory. We use the DB methods to perform the scan and query operations. The key aspect of this setup is that, you don’t need to go through any implicit data serializtion or de-seralization, and those things are handled behind the scenes consistently with the use of the defined TableSchema’s

App In Action

The controller is just going to relay the requests to the service layer

package com.itassistors.books;

import com.itassistors.books.entity.Book;
import lombok.extern.slf4j.Slf4j;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.*;

import java.util.List;

@Slf4j
@RestController
@RequestMapping("/api/v1")
public class BookController {

    private final BookService bookService;

    public BookController(BookService bookService) {
        this.bookService = bookService;
    }

    @GetMapping("/books")
    public List<Book> getBooks() {
        log.info("Fetching all books");
        return bookService.getAllBooks();
    }

    @PostMapping("/book")
    public Book addBook(@RequestBody Book book) {
        log.info("Adding a new book: {}", book.getTitle());
        return bookService.addBook(book);
    }

    @GetMapping("/book/{id}")
    public ResponseEntity<Book> getBookById(@PathVariable("id") Long id) {
        log.info("Fetching book by id: {}", id);
        Book bookById = bookService.getBookById(id);
        if (bookById == null) {
            log.error("Book with ID {} not found", id);
            return ResponseEntity.notFound().build();
        }

        return ResponseEntity.ok(bookById);
    }

    @GetMapping("/books/year/{yearPublished}")
    public List<Book> getBooksByYearPublished(@PathVariable("yearPublished") Integer yearPublished) {
        log.info("Fetching books published in the year: {}", yearPublished);
        return bookService.getBookByYearPublished(yearPublished);
    }
}

Conclusion

Using the EnhancedClient API solves the major pain-point of data serialization and it also enforces a strict, yet a flexible schema for the tables. One might find it boring to write the TableSchema for the beans, and I’m one. So I delegate the task of generating the table schemas to copilot. ChatGPT also does a better job of doing this provided you give it a tailored prompt

Happy Coding!

Not pre-signed URLs | Share Objects from Private S3 Buckets

Neel — Thu, 29 Feb 2024 08:23:00 +0000

For those who have wandered the path already, No, it is not just pre-signed URLs

You are a budding developer who has just started learning the concepts of AWS. After you have tinkered with those EC2 instances and levied an unaffordable amount on your AWS invoice, you need a break. So you resort to the most friendly AWS service of all, the Simple Storage Service (S3). You open the console, you create a bucket with a unique name, you keep the bucket private, you add the bucket policies (if you are savvy enough) and then you save and start uploading a bunch of screenshots to see if everything has gone as per plan.

If you have a bucket, then it is known that the bucket is going to be a private one. Exposing the bucket to the public is a task meant for rugged souls, not us developers. If you have a private bucket, then sharing the objects from the bucket is going to be a bit challenging. You can't just copy the object's URL and share it around. Opening it will show an XML error response stating that you are not allowed to access the resource. After Googling for a bit, you find out that there is this thing called a "pre-signed URL" which does the trick for you and you directly take that route. I have been there and I have done that, and this is a story of how I found out that pre-signed URLs are not a one-stop solution for every use case

Presigned URL

Presigned URLs are the simplest and a secure way to share objects from a private S3 bucket. These URLs are secured and short-lived which makes them perfect for sharing individual objects with desired parties. The process of generating a pre-signed download URL is straightforward and the screenshot below shows how it can be done from the AWS console, CLI, and using the SDK

The resultant URL will be in the following format

https://s3.{region}.amazonaws.com/{bucket name}/{object key}?
    X-Amz-Algorithm=AWS4-HMAC-SHA256&
    X-Amz-Credential={credentials generated by AWS}&
    X-Amz-Date={UTC datetime}&
    X-Amz-Expires={expiry in seconds}&
    X-Amz-Signature={unique signature}

There are a few good things about this URL,

Its validity is limited. If you set the expiration value to 3600 seconds then the URL will be invalid after an hour and the object will become inaccessible after that
The URL is read-only. Once the pre-signed URL is generated, none of its parameters are editable. No one can try to edit the object key or the expiration value from the URL to do any funny stuff. Modifying any part of the URL will make it invalid
The URL is linked to the user generating it and inherits the policies assigned to the user. By controlling the policy, the usage of the URL can be restricted

All the above points make a pre-signed S3 URL a viable option for many use cases. However, the same advantages might turn out to be drawbacks in certain scenarios. For instance,

The pre-signed URLs cannot have an expiration limit of more than 7 days. So if your use case involves making it accessible for more than 7 days, then it will not be possible
The pre-signed URLs generated by a user are bound to their credentials. This means that when the credential becomes invalid, the URL becomes invalid too. This will make more sense when you use a lambda function to generate these URLs

What was my use case?

Here is my story of trying out a pre-signed URL and moving away from it, as it did not suit all my needs.

The AWS lambda function I was building was supposed to collect a bunch of PDF documents stored in an S3 bucket, generate an HTML email with some details, and include the download links for all the documents in the email sent to the recipients. Simple, init it?

Failed Approaches

Using a Lambda function with an execution role

The first pothole I hit was with the validity of these URLs generated using the lambda function. You can assign an execution role to a lambda function and it will be able to access only the resources that you have defined for the role's policy. This ensures that the lambda functions will not be allowed to perform any unintended activities. A sample IAM policy defined for an execution role will look something like this 👇

The above policy ensures that the function can only get, put, and delete objects from the selected S3 bucket. The execution role is not bound to any real IAM user, so to perform any action, the lambda creates temporary credentials to access the resources. The generated temporary credentials are bound to a short-lived session and the default session duration is 1 Hour

No matter what the maximum session duration is, when used outside the scope of the lambda function such as to generate a pre-signed URL, the session cannot be controlled. For instance, if the temporary credentials used to generate the URL expires, then in affects the validity of the URL too, thus making the option of generating a pre-signed URL using lambda function non-viable

Using a Lambda function with IAM credentials

After discovering this limitation, I knew that there was another way to do it with lambda functions. This is not new or innovative in any way, because when you are generating pre-signed URLs using AWS CLI, then you would notice that the URLs will stay valid for a week (provided you set the expiry to a week). This makes sense because we "mostly" configure the access key and secret of an actual IAM user to use the CLI. So the URLs generated using those credentials will be valid for up to 7 days

🚨⚠️ Approach with Caution ⚠️🚨

The same can be done using lambda functions by creating a new IAM user (using CDK/Cloudformation is recommended)
Storing the access key & secret as a new secret value in the AWS secrets manager
Creating a new S3 client by using a StaticCredentialsProvider that uses the access key and secret fetched from the secrets manager

This will do the trick but this is not without its drawbacks. First of all, it is not a good idea to use statically configured credentials to access AWS resources via lambda functions and the IAM user could go out of control if the access is shared for local access or other purposes. If you are working alongside a good Cybersecurity team then they might go berserk hearing about this approach

So this method is definitely out the window with no second thoughts 🤷‍♂️

The one that worked!

Using CloudFront signed URL

The final solution that perfectly suited my requirements came in the form of CloudFront signed URLs

In Amazon's own words, "Amazon CloudFront is a fast content delivery network (CDN) service that securely delivers data, videos, applications, and APIs to customers globally with low latency and high transfer speeds"

I started using CloudFront to deliver my static website hosted in an S3 bucket to users located in different regions of the world. It caches the resources at the nearby edge servers thus providing low latency access to the users, no matter where they are located

This means that you can serve any content from your S3 bucket using CloudFront and custom settings can control the access level of the resources.

The process is not simple but it is approachable. Let's see how it is done

Creating a new CloudFront distribution

The first step is to create a new CF distribution. As it is a CDN that works across different regions, the service is not bound to any single region (such as us-east-1 or ap-south-1). It is available globally. Search for the "CloudFront" service and click on "Create a CloudFront distribution" to set up a new distribution

The above is the creation form and the purpose of each field is as follows,

Fields	Purpose
Origin domain	The source of your content and this input field will display all the available S3 buckets from your account. You can select the bucket from where you wish to serve the content
Origin Access	The access boundary for the S3 bucket Public - Provided the bucket has public access, this option will allow anyone to access the objects from the bucket using the generated CloudFront URL (e.g. https://custom.cloudfront.com/image.png) Origin access control settings - This is the recommended access level and we will use this to create a new control setting that enables only signed URLs to access the content
Restrict viewer access	This will control the viewing access of the content from the bucket No - Anyone can download/view the content using the custom CloudFront URL (e.g. https://custom.cloudfront.com/image.png) Yes - The objects will be accessible only using a signed URL (or a signed cookie). This is the recommended setting

The above three are the notable fields. You can follow the input selection from the above screenshot to create a bare distribution. Remember that we will come back to the "Origin Access" and "Restrict viewer access" options to make the entire setup secured

Once you are done with the creation, you will land on the distribution page where you will see the custom domain setup for the new distribution. It can be used to access the content from the S3 bucket, but not yet

Create a Key Pair

Now is the time to create a new key pair using openssl. We will generate a private and public key from the terminal and configure the public key to CF. If you are using a Mac, then openssl can be installed using brew. It will be available on some of the Linux distros, I believe. If not, then it can be installed using the official package manager. For Windows, if you have Git bash installed then it comes bundled with that

Run the following commands to generate a private and public key pair

# Generates the private key for the server
~ openssl genrsa -out private_key.pem 2048

# To view the private key
~ cat private_key.pem

--------BEGIN PRIVATE KEY-----
RANDOMRANDOMRANDOMRANDOM
...
--------END PRIVATE KEY-----

# To extract the public key from the private key
~ openssl rsa -in private_key.pem -pubout -out public_key.pem

# To view the public key
~ cat public_key.pem

--------BEGIN PUBLIC KEY-----
RANDOMRANDOMRANDOMRANDOM
...
--------END PUBLIC KEY-----

Storing the Private Key as a Secret

Open the generated private key file using a text editor or in the terminal and copy the entire content of the file. Open Secrets Manager from the AWS console and click on "Store a new secret" to create a new secret with the private key value.

Choose "Other type of secret" as the secret type and paste the copied private key into the "Key/value pairs" field as a Plaintext
In the next section, give a valid secret name (e.g clodufront/privatekey) and leave the rest of the fields to default, and save the secret

#CLI command to create a new secret from the private key file
~ aws secretsmanager create-secret \
    --name "clodufront/privatekey" \
    --secret-string file://private_key.pem

Create a new CloudFront Public Key and Key Group

From the CloudFormation console, open the "Public Keys" section from the sidebar. Click on "Create public key" and paste the entire content copied from the public_key.pem file and save it

# Create a CloudFront public key using the CLI
aws cloudfront create-public-key \
    --public-key-config \
    "{\"CallerReference\":\"cli-1\",\"Name\":\"testkey\",\"EncodedKey\":\"-----BEGIN PUBLIC KEY-----\nMIIBI<....>QAB\n-----END PUBLIC KEY-----\n\"}"

After creating the new public key, open the "Key Groups" section from the sidebar. Create a new key group by selecting the newly added public key from the dropdown

aws cloudfront create-key-group \
    --key-group-config \
    '{"Name":"NewKeyGroup","Items":["K271M92XK8HX1O"]}'

Origin Access Control and Viewer Access Restriction

Remember the two fields from the table above which we discussed to revisit later? Well, it is time to do that

Let us start with creating a new Origin Access Control setting. Open the new distribution you have created and navigate to the "Origins" tab from the console. Select the only origin from the list and click "Edit".

On the edit page, change the "Origin access" selection from "Public" to "Origin access control settings"
On selecting this option, you will see a new action button to "Create control setting". Click this to open the popup that will let you create a new setting

Give the setting a name and select the signing behaviour to "Sign requests". This will make CloudFront create a new signature for every HTTP request that comes in and this signature will be forwarded to S3 to get the actual content. If the signature validation fails, then the content will not be served
Set the "Origin type" to "S3" and save the setting
After saving the control setting, you will see an option to update the S3 bucket policy. This is to allow the CloudFront distribution to fetch the content from the bucket for which it needs the "s3:GetObject" action to be configured. Click the "Copy policy" button to copy the exact policy that needs to be configured and click on the "Go to S3 bucket permissions" to update the bucket policy with the copied content
After saving the bucket policy, come back to the "Edit origin" page, leave the rest of the fields as it is, and save the changes

We are not done yet. There is one final bit that completes the entire circle. From the distributions page, navigate to the "Behaviors" tab. Select the default behaviour from the list and click "Edit"

On the edit behaviour page, scroll down to find the "Restrict viewer action" input and change the option to "Yes"
This will show you an option to select the "Trusted authorization type" and from the inputs, select "Trusted key groups"
From the key group dropdown, select the new group that you have set up a while ago
Once the group is selected, leave the other fields as it is and save the changes

Moment of truth

If you followed along so far and applied all the changes then you are close to the ultimate goal. We will see how we can generate a signed URL using Python and Java Lambda functions

Setup the following as the environment variables for the lambda functions

Variable	What is it?
CLOUDFRONT_DOMAIN	The domain of the created distribution
CLOUDFRONT_PUBLIC_KEY_ID	The unique ID of the public key. This can be found under the "Public Keys" section in the CloudFront sidebar e.g. K1X0XXXXXXXXXX
SECRET_ID	The name of the secret that holds the private key. In this article, I have set it as "clodufront/privatekey"

Python Lambda Function

Java Lambda Function

The Ultimate "Signed URL"

Both these functions will return the signed URL as a string that can be used to access a resource from the S3 bucket. The following are a few highlights about this URL,

It is not bound to the Lambda's execution role, so it is not dependent on the temporary session
In the above examples, the expiry is set to 7 days but it can be extended beyond that unlike the S3 pre-signed URL
The signed URL depends on the Key Pair which can be rotated easily whenever needed

Conclusion

It took me a couple of days to find this solution, so I wanted to share the same here. Hope this article helped a few.

As always, Happy Hacking!☕

Document search tool for Github repos built with React, Supabase and LLAMA2

Neel — Tue, 10 Oct 2023 18:10:38 +0000

Building document Q&A apps using LLMs has become mainstream now and I built one a few months ago. Constructing a prompt with a context and getting a response from an LLM is pretty simple. Tools like the OpenAI library and the huggingface transformer library have made it a walk in the park. The fun part of building such apps is putting together the different layers of the application to generate a proper prompt with a context, that gives you a near-accurate response out of the LLM. So, this article will focus more on those layers and techniques rather than showing the complete source code for every operation. Because the code is pretty simple and it is the interaction between the different layers that makes this more interesting. Anyway, the entire project is available on Github for you to explore and hack out something creative

I have used supabase extensively for building this application and it being a super awesome platform, has done a lot of heavy lifting with clean abstractions. The project uses supabase for storage & retrieval, authentication, real-time notifications and on-the-fly embedding generation

Apart from that, it is a simple react app, making a bunch of api calls and mutating the components on the screen

Prerequisites

A react app bootstrapped using vite
A Supabase account with a new project set up. The project should include a few tables which we will discuss in the upcoming sections
A huggingface account to download the LLAMA2 model
A little bit of Python knowledge to write the inference API

What does this app do?

The app is a simple search tool for extracting content out of the documentation files that you have in your github repos. If you are working for an organization that uses github then there could be a bunch of documents stored in the form of Markdown files (.md), .mdx files or .rst files. These files contain valuable information such as setup instructions for a project or how running a single script within the project could drop the entire production database or something vital. This app lets you get a specific piece of information from such documents using conversational search queries. You can ask it "What is the request schema for creating a new user?" and it will give you the relevant output by gathering information from your documents

Below is a sample

Github REST API

After going through all the words from above, you must have gotten a rough idea that we need to collect information from your github repositories. The content we need includes the repository info, the list of files from the repo and the actual content of the files

You would have already guessed that we are not going to do all these things manually by opening every markdown file and copying the content out of it. As fun as it sounds, it could be time-consuming. So we will be using the REST api provided by github to do everything, just like normal devs would do. If you want to access private repos then you need to be authenticated

If you are just testing out the API or using it for a simple use case, then a simple Personal Access Token would suffice, but for an app that could be used by multiple users, it is not recommended to use a common auth token. The clean way is to use a proper auth setup and that is where supabase authentication comes into play

Supabase authentication

Using Supabase Authentication, we can set up a new github oauth provider that can be used to authenticate the users. This lets you log in to the app using your github credentials and granting access to selective orgs that you are part of. Based on your selection, only the granted org repositories will be available to you when you use the REST api

Setting up the github provider is pretty simple and the following are the steps,

The first step is to create a new Github oauth app
On creating the oauth app, you will get a client ID and a secret, and the same needs to be configured under the supabase auth provider section

The following documentation explains the entire process clearly and it can be referred to get a clear picture

https://supabase.com/docs/guides/auth/social-login/auth-github

In the react app, you can use the signInWithOAuth function to login with the github oauth app. We specify the scope as "repo" and this will limit the access of the user to the repository-oriented operations alone. On signing in, supabase will automatically persist the session information in the browser's localStorage and you can fetch the session anytime you want using the supabaseClient.auth.getSession() method

supabaseClient.auth.signInWithOAuth({
  provider: "github",
  options: { scopes: "repo" },
});

The session data includes the user and identity information, and the important thing that we need is the provider_token. This is the temporary auth token provided by github and we can pass this as the bearer token to access the Github REST API endpoints. Without this token, you will get access only to the public repos from the org and the private repos will be out of scope

In the app I have created a react hook that calls the getSession() method to fetch the provider_token and return it to the components/hooks that invoke it. You can use your favorite HTTP client for calling the api endpoints. I use axios in this project with a common instance containing the baseURL and all the required request headers, so that I need not duplicate it in multiple places. Below is a sample code to fetch the existing user session

return supabaseClient.auth.getSession().then(({ data }) => {
  if (!data.session) {
    return null;
  }

  if (!data.session.provider_token) {
    return null;
  }

  return {
    accessToken: data?.session.access_token,
    providerToken: data?.session.provider_token,
    refreshToken: data?.session.refresh_token,
    userName: data?.session.user.user_metadata.user_name,
  };
});

Ingesting the document content

Now that we are done with the auth setup, the next stage is to ingest the content from all the valid documents into the Database. This content will be supplied as the context in the prompt for chatting with LLAMA2

Table creation

We need to create 3 tables in supabase and the purpose of those tables are as follows

repositories - To store the repository and the org data
documents - To store the document path, content and related repository ID
document_embeddings - To store the embedding vector and the content of the documents. The document_embedding is a vector field and this needs the pgvector extension to be enabled for your supabase project.

When the user lands on the application, they will get the following page. This shows all the repositories that are part of the org and the ones that the user is authorized to access

Below are the steps to get the repo list like the one we have above,

Fetching the repos

As the repos are part of the org, we must fetch the list of orgs first and then use the org names to fetch the repo list

/users/{user_name}/orgs - This endpoint returns the org list. The user name in the path can be obtained from the supabase session data session.user.user_metadata.user_name. The response will be an array with the list of repos and the login field in the JSON denotes the name of the org

/orgs/{org_name}/repos - Invoke this endpoint iteratively with the org names to fetch the repositories and extract only the required fields from the response. For this application, we need just the repository name which will be available as the name field in the response body. To keep track of the repositories, we check the repositories table from the DB and if a record does not exist, then we will insert it into the table

const getAllRepositories = async () => {
  const { data, error } = await supabaseClient.from("repositories").select("*");

  if (error) {
    console.error(error);
    return null;
  }

  if (!data || data.length === 0) {
    return null;
  }

  return data;
};

// Compare the repo list response from the Github api
// If the table does not have a matching record, then insert the repo into the table
const insertNewRepository = async ({ org, repo }) => {
  const { data, error } = await supabaseClient
    .from(table)
    .insert({ name: repo, org })
    .select("*");

  if (error) {
    console.error(error);
    return null;
  }

  if (!data || data.length === 0) {
    return null;
  }

  return data[0];
};

The repository details that we are adding to the DB will be used in the actual document ingestion stage

Once all the record collection is done, you can render a clean view like the one above. In the actual application, I have used @tanstack/react-query and a few react hooks to fetch the response from the API. The table is rendered using @mui/x-data-grid

Fetching the list of files

The next step is to get the document files from the repository. When the user clicks on the repo from the above table, the app navigates the user to a new page. This page includes the basic repo details and the filtered documents

The repo details that you see on the page are fetched from the DB table and the documents are fetched from the Github REST api

To get the files from a github repo,

You need to fetch the default branch of the repo
Use the branch to fetch the files from the HEAD of the repo

/repos/${org}/${repo_name} - This will give the repo data and the response includes a field called default_branch which is the default branch of the repository

/repos/${org}/${repo_name}/git/trees/${default_branch}?recursive=1 - This endpoint will list all the latest version of the files from the repo's default branch. The response includes a field called as tree which is an array of all the files in the repo. The path field within the trees array has the absolute path of the files. This array needs to be looped through to filter only the document files such as *.md, *.mdx etc., It is just a matter of splitting the path extension and including it only if it matches the allowed extensions

const allowedExtensions = ["md", "mdx", "rst"];

const filteredFiles = response.data.tree
  .filter((item) => {
    const extension = item.path.split(".").pop();
    return allowedExtensions.includes(extension);
  })
  .map((item) => {
    return {
      documentPath: item.path,
    };
  });

Now you can use the filtered document names to render a table with the status and an action button to ingest it

Downloading and ingesting the documents

Now that we have all the documents, we can manually ingest a single document or all the documents from the repo to the Database. Ingesting is nothing but downloading the content of the document, inserting it into the DB and generating the embedding vector for the content

We will use the serverless prowess of supabase to handle all these

Downloading the document content - The content of a file can be downloaded using the /repos/${org}/${repo_name}/contents/${path} endpoint. This will return the content of the file in base64 format and we can convert it into utf-8 using Buffer.from(response.data.content, "base64").toString()

Saving the content - Store the entire document content in the documents table along with the other required fields (document_name and repo_id)

Generating Embeddings - If you are new to LLMs or embeddings, then I suggest going through this article from supabase. It explains the concepts clearly and how you can use supabase to store the embedding vector. To generate the embeddings for the documents, we will make use of supabase edge functions and webhooks to do it in an event-driven fashion

Supabase edge functions use deno as the runtime and you can learn more about it here. We create a new function using the supabase CLI and it generates an index.ts file which will serve as the entry point for the function. Inside this file, we will write the logic to generate the embeddings using the huggingface transformers library. A basic version of the function will look something like this. If you want the entire code, then it is available here

import { serve } from "https://deno.land/std@0.168.0/http/server.ts";
import {
  env,
  pipeline,
} from "https://cdn.jsdelivr.net/npm/@xenova/transformers@2.5.0";
import { createClient } from "https://esm.sh/@supabase/supabase-js@2";

env.useBrowserCache = false;
env.allowLocalModels = false;

const supabaseClient = createClient(
  Deno.env.get("SUPABASE_URL") ?? "",
  Deno.env.get("SUPABASE_ANON_KEY") ?? ""
);

const pipe = await pipeline("feature-extraction", "Supabase/gte-small");

serve(async (req) => {
  const data = await req.json();

  const {
    record: { id, document_name, document_content, repo_id },
  } = data; // This payload will be sent by the webhook

  const output = await pipe(document_content, {
    pooling: "mean",
    normalize: true,
  });
  const embeddings = Array.from(output.data);

  const payload = {
    document_id: id,
    document_name,
    embeddings,
    document_content,
  };

  // Inserting the generated embeddings into the table
  const res = await supabaseClient
      .from("document_embeddings")
      .insert(payload);

  return new Response(JSON.stringify({ payload }), {
    headers: { "Content-Type": "application/json" },
  });
});

This documentation from supabase explains this process clearly and the use of each of every line from the function

After deploying the edge function, we need to setup a webhook and that can be done from Supabase project -> Database -> Webhooks. The webhook will be set for all INSERT and UPDATE events on the documents table and the trigger will be the edge function that we have just created. Whenever we insert a new document or update an existing document, the webhook will trigger the edge function to generate the embedding vector for the document content

Once the edge function completes the process, we will have the table with the embedding vector thus completing the entire ingestion flow. To ingest all the documents from repo, we just need to repeat this process iteratively and the app includes an "INGEST ALL" button which does the same

As an extra flare, I have used supabase realtime to notify the UI about the ingestion status. On clicking the "INGEST" action button, the status gets toggled to "Processing" and the edge function will notify the completion status to update the status to "Ingested"

Searching the documents

We have everything in place now, which means we can pass it on to the LLM and get a response

Why llama2? Well, The first reason is that I wanted to experiment with the model and the next is that, I thought it would be cool to tell people that I have an LLM running on my personal computer. In addition to that, if you are building a similar application for your team and if your team is skeptical about passing around sensitive data to an externally hosted LLM service, then using an open LLM model hosted on your org's infrastructure will be a viable alternative. Keep in mind that this is a Large Language Model (emphasis on the word "Large") and running it even for inference alone is going to be computationally intensive and you need a powerful machine to pull it off. Let's just say that you won't get favorable results by running the model on a t2.micro instance

To be specific, the project uses the GGUF variant of LLAMA2 which is the quantized version of the original model. In theory, quantized models are lightweight hence they will be less resource intensive. You can read more about it here

The steps involved in chatting with the model are as follows,

Similarity search - When the user enters the conversational query to search the document, we need to perform a vector similarity search to fetch only the relevant document content. This can be done using supabase DB functions.

You can execute the following query in the supabase SQL editor and it will create a new function for you that will perform the similarity search on the embeddings to filter only the closely related content

create or replace function match_documents (
  query_embedding vector(384),
  match_threshold float,
  match_count int
)
returns table (
  document_id uuid,
  document_content text,
  similarity float
)
language sql stable
as $$
  select
    document_embeddings.document_id,
    document_embeddings.document_content,
    1 - (document_embeddings.embeddings <=> query_embedding) as similarity
  from document_embeddings
  where 1 - (document_embeddings.embeddings <=> query_embedding) > match_threshold
  order by similarity desc
  limit match_count;
$$;

From the UI we need to generate the embedding vector for the user query and the above function will perform the similarity search based on the generated vector to return the relevant content

import { pipeline } from "@xenova/transformers";

const pipe = await pipeline("feature-extraction", "Supabase/gte-small");

const generateEmbeddings = async (query) => {
  const output = await pipe(query, {
    pooling: "mean",
    normalize: true,
  });

  return Array.from(output.data);
};

const search = async (query) => {
  const { data, error } = await supabaseClient.rpc("match_documents", {
    query_embedding: generateEmbeddings(query),
    match_threshold: 0.5,
    match_count: 2,
  });

  return data
    .map((item) => {
      return item.document_content;
    })
    .join(" ");
};

Inference API - This is the only layer where we burst out of the JS bubble and take a stroll into the Python land. We will create a very simple Python script to run the inference on LLAMA2 and expose it as a REST endpoint using FastAPI

from fastapi import FastAPI
from pydantic import BaseModel
import uvicorn
from ctransformers import AutoModelForCausalLM

app = FastAPI()

model_id = "TheBloke/Llama-2-7b-Chat-GGUF"
llm = AutoModelForCausalLM.from_pretrained(
    model_id, context_length=4096, model_type="llama", gpu_layers=120
)

class InferenceRequest(BaseModel):
    query: str
    context: str

class InferenceResponse(BaseModel):
    response: str

@app.post("/api/infer")
async def inference(request: InferenceRequest) -> InferenceResponse:
    prompt = f"""
        [INST] <<SYS>>
        Use the following pieces of context to answer the question at the end. If the question asks to list something, then list it as Markdown bullet points with a newline character. If the answer includes source code or commands, then format it using Markdown notation. If you don't know the answer, truthfully say "I don't know". If the question is out of the context, then say "The query is not related to the context".\n\nContext:
        {request.context}
        <</SYS>>

        User: {request.query} [/INST]

    """

    generated_text = llm(prompt, max_new_tokens=4096)

    return InferenceResponse(response=generated_text)

if __name__ == "__main__":
    uvicorn.run(app, port=8000)

The above api accepts the user query and the filtered document content as request payload, using which we generate a prompt. Notice that we have a bunch of gibberish in the template like [INST],<<SYS>>. This is the suggested way of writing a prompt for LLAMA2 and more details about the purpose of these notations can be found in this blog

Once this API is invoked, the prompt is generated and passed to the pipeline. It takes a few seconds or minutes to respond with the answer depending on the system specifications. The model runs better on a GPU and the gpu_layers=n parameter runs the specified number of layers in the GPU

I haven't run a thorough benchmark, but to give a rough estimate, the inference takes somewhere around 20 to 30 seconds to complete on my machine powered by an RTX 3060 12G GPU. If you have a more powerful GPU, then the performance will be even better

By invoking this API from the react app, we can get the relevant results as the response.

That's all, We did it

That's all to it. This is how you build a document search tool for your github organization using React + Supabase + LLAMA2. This use case is not specific to github alone and you can use a similar setup for platforms like Confluence, Sharepoint or Google Docs. The API you use to fetch the content will differ but the embedding generation, similarity search and LLM inference will remain the same

I initially tried to run the LLM within the browser inside a worker thread and tranformers.js enables this using ONNX models, but I couldn't get it to work with LLAMA2 and that is something I will be working on in the future

The entire project is available on Github. Configure the auth tokens for Supabase & Huggingface, and try it out

Thanks for reading and Happy Hacking!

References

You don't need sockets to chat with GPT

Neel — Sat, 16 Sep 2023 11:07:03 +0000

I am not sure if it is just the case with me or everyone else. Whenever I hear the word "chat", my brain immediately pops up a light bulb that says "Sockets! You need sockets". I recently built a chatbot powered by GPT and after working with it for a while, I realized that you don't need sockets to chat with GPT. To put it into perspective, ChatGPT does not use sockets if you take a peek under the hood

Sockets are good for two-way non-blocking communication, where both the actors on scene act as the sender and the receiver. If you are building a chatbot powered by GPT (or any other model from OpenAI), then you don't need a socket server/client. Because till now, GPT has not initiated any conversations (Let's hope it doesn't... Ever!). As a user, you initiate the conversation every time and you wait for the response from the model

In the above clip, the concept you are witnessing is called Server-Sent Events. The client makes a simple REST api call to open a persistent connection with the server and the new messages from the server will be sent in the form of events to the open connection. In the case of ChatGPT, instead of keeping the user waiting till the model returns the entire response, the api streams the message chunks in the form of events, as and when they become available. This gives a feel to the user that they are not waiting forever to get a response from the server

We will build a simple chatbot using react and OpenAI client library for JS, and then we will see how the same can be achieved with a Node JS backend. We will use a simple REST endpoint in the backend and relay the response data to the UI without using any socket implementations.

⚛️ Client-side Implementation (React + OpenAI library )

I have used Vite to bootstrap a new React app and installed the openai dependency alone

yarn create vite #to create a new react app using vite

yarn add openai #installing the openai client library as a dependency

I don't want to deviate from the topic by installing a ton of other things, so I will keep it simple. We will just build a simple chat UI using vanilla react and inline styling

OpenAI client setup

We will get started by creating a new instance for the OpenAI client library

import OpenAI from "openai";

const openai = new OpenAI({
  apiKey: import.meta.env.VITE_OPENAI_API_KEY, //Store this in your .env file
  dangerouslyAllowBrowser: true,
});

export { openai };

OpenAI guidelines specifically discourage the use of their API keys on the client side. However, if you understand the risk then you will be allowed to use it on the client side by setting the dangerouslyAllowBrowser field to true

Components

The next step is to create a component to wrap all the chat elements. I will make use of the default App.jsx file as the parent to hold all the components

/* App.jsx */

import { useState } from "react";
import { openai } from "./openai";
import InputForm from "./InputForm";
import Conversations from "./Conversations";

export default function App() {
  const [conversations, setConversations] = useState([]); // To track all the conversations
  const [aiMessage, setAiMessage] = useState(""); // To store the new AI responses
  const [userMessage, setUserMessage] = useState(""); // To store the message sent by the user

  const handleSubmit = useCallback(
    async (e) => {
         // We will handle the user message submission and streaming here
    },
    []
  );

  // The app will have a vertical layout with the conversation section at the top
  // The message textbox will be at the bottom
  return (
    <div
      style={{
        width: "100%",
        height: "100%",
        display: "flex",
        flexDirection: "column",
        justifyContent: "center",
        background: "darkslateblue",
      }}
    >
      {/* Conversations section */}
      {/* Message input */}
    </div>
  );
}

The above is the wrapper that will contain the chat bubbles on top and the message input field at the bottom. Now we will create a new component that will hold a form to accept the user messages

/* Message input */
/* InputForm.jsx */

export default function InputForm({
  userMessage,
  setUserMessage,
  handleSubmit,
}) {
  return (
    <form
      style={{
        width: "60%",
        display: "flex",
        flexDirection: "row",
        justifyContent: "center",
        background: "white",
        margin: "auto",
      }}
      onSubmit={handleSubmit}
    >
      <input
        type="text"
        placeholder="Ask something..."
        value={userMessage}
        onChange={(e) => setUserMessage(e.target.value)}
        style={{
          width: "90%",
          outline: "none",
          padding: "20px",
          border: "none",
        }}
      ></input>
      <button
        type="submit"
        style={{
          width: "10%",
          outline: "none",
          padding: "20px",
          border: "none",
          background: "mediumslateblue",
          color: "white",
          fontWeight: "bold",
          cursor: "pointer",
        }}
      >
        Send
      </button>
    </form>
  );
}

The resultant UI will look like this. Neat, isn't it?

Now that we have an input form, we will proceed to create a basic chat bubble component and a scrollable section to display all the chat bubbles with the messages from the user and the AI

/* ChatBubble.jsx */

export default function ChatBubble({ isHuman, message }) {
  return (
    <div
      style={{
        width: "100%",
        display: "flex",
        flexDirection: "row",
        justifyContent: isHuman ? "flex-end" : "flex-start",
      }}
    >
      <div
        style={{
          width: "fit-content",
          maxWidth: "600px",
          background: isHuman ? "lightblue" : "white",
          padding: "10px",
          borderRadius: "10px",
          margin: "14px",
          color: "black",
          fontFamily: "sans-serif",
          fontWeight: "400",
        }}
      >
        {message}
      </div>
    </div>
  );
}

If the message is from the user, then it will float on the right. The messages from the AI will be on the left. The below is the look we are going for

Now the scrollable section. This will be the first child of the wrapper component

/* Conversations section */
/* Conversations.jsx */

import ChatBubble from "./ChatBubble";

export default function Conversations({ conversations, aiMessage }) {
  return (
    <div
      style={{
        width: "100%",
        height: "90%",
        display: "flex",
        flexDirection: "column",
        justifyContent: "flex-end",
        margin: "0 auto",
        background: "#625a91",
      }}
    >
      <div
        style={{
          maxHeight: "800px",
          overflowY: "auto",
          padding: "30px",
        }}
      >
        {conversations &&
          conversations.map((conversation, index) => {
            return (
              <ChatBubble
                isHuman={conversation.isHuman}
                message={conversation.message}
                key={index}
              />
            );
          })}
        {aiMessage && <ChatBubble isHuman={false} message={aiMessage} />}
      </div>
    </div>
  );
}

Now, with the entire component put together, the UI will look as below,

The UI-related things are all done now. The one major part that is pending is the actual thing for which you opened this article in the first place. THE STREAMING!

Streaming the response

We have created a function within the App component to handle the message submission from the user. We will handle the logic of invoking the OpenAI api and streaming the response. The responsibilities of the submission handler are as follows,

To store the user message in the global conversations state and empty the chat input field
To invoke the OpenAI api with the user message and stream the response
To store the AI response to the conversations state once the stream is closed

const handleSubmit = useCallback(
    async (e) => {
      e.preventDefault(); //To prevent the default form submission behaviour

      // Storing the user message to the state
      setConversations((prev) => {
        return [
          ...prev,
          {
            message: userMessage,
            isHuman: true,
          },
        ];
      });
      setUserMessage(""); // Emptying the input field

      const stream = await openai.chat.completions.create({
        model: "gpt-3.5-turbo",
        messages: [{ role: "user", content: userMessage }],
        stream: true, // This is required to stream the response
      });

      let streamedMessage = "";

      // The stream will be read till its closed
      for await (const part of stream) {
        setAiMessage((prev) => prev + part.choices[0].delta.content);

        // Once the entire message is received, the stream will receive the finish_reason as 'stop;
        if (part.choices[0].finish_reason === "stop") {
          setConversations((prev) => {
            return [
              ...prev,
              {
                message: streamedMessage,
                isHuman: false,
              },
            ];
          });

          setAiMessage("");
          break;
        } else {
          streamedMessage += part.choices[0].delta.content;
        }
      }
    },
    [userMessage]
  );

The above handler will take care of the streaming and storing part. If you notice, we have a temporary state to store the AI message. This is to avoid mutating the conversations state (which is an array), multiple times based on frequently streamed data. So once the entire stream is closed, we will reset the temporary state and store the message in the conversations list

Demo

With everything put together, the app will do what it was intended to do

It streams the response and mutates the UI. This gives a feel that the user is not waiting for a long time to get a response, rather they get visual feedback as and when chunks of response are available

This concludes the client-side setup for streaming response from OpenAI

🌐 Backend Implementation (Node JS + Express + OpenAI library)

Why server? Well, as I mentioned above, OpenAI doesn't want you to use the apiKey on the client side and they have a very strong reason for it. We will create a new node js project and install the required dependencies

mkdir server && cd server
yarn init -y
yarn add express openai

Route to handle conversations

We will create a file named server.js and set up everything required within this single file. It is pretty straightforward with express

import express from "express";
import OpenAI from "openai";

const app = express();

const openai = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY, //Ensure this is a part of your env variables
});

app.use(express.json());

app.post("/api/chat", async (req, res) => {
  const { message } = req.body;

  const stream = await openai.chat.completions.create({
    model: "gpt-3.5-turbo",
    stream: true,
    messages: [
      {
        role: "user",
        content: message,
      },
    ],
  });

  for await (const part of stream) {
    const reason = part.choices[0].finish_reason;
    if (reason === "stop") {
      break;
    }
    const chunk = part.choices[0].delta.content;
    res.write(chunk); //Write each chunk to the open stream
  }

  res.end(); //Close the connection once the streaming is done
});

app.listen(5000, () => {
  console.log("Server is listening on port 5000");
});

In the above server-side code, the route /api/chat will handle the communication with the OpenAI api. We send the message from the UI in the request body and pass it over to the chat completion api. We use the res.write(chunk) function to continuously stream the data back to the UI.

Read from the stream in the UI

We need to have a new reader in the UI to read the streamed data and mutate the state accordingly. The handleSubmit function with the logic to read from the server response will look like this,

const handleSubmit = useCallback(
    async (e) => {
      e.preventDefault();

      setConversations((prev) => {
        return [
          ...prev,
          {
            message: userMessage,
            isHuman: true,
          },
        ];
      });
      setUserMessage("");

      const res = await fetch("/api/chat", {
        method: "POST",
        headers: {
          "Content-Type": "application/json",
        },
        body: JSON.stringify({ message: userMessage }),
      });

      if (!res.ok) {
        console.log("Error in response");
        return;
      }

      let streamedMessage = "";
      const reader = res.body.getReader();

      while (true) {
        const { done, value } = await reader.read();
        if (done) break;

        const decoded = new TextDecoder("utf-8").decode(value);
        streamedMessage += decoded;
        setAiMessage((prev) => prev + decoded);
      }

      setConversations((prev) => {
        return [
          ...prev,
          {
            message: streamedMessage,
            isHuman: false,
          },
        ];
      });

      setAiMessage("");
    },
    [userMessage]
  );

We get the reader from the response body returned by fetch and we should keep reading the data till the stream is closed. When the stream is closed, the value of done will be true and use that to break out of the loop (Ensure this is done, otherwise the loop becomes infinite). The data read from the stream will be an unsigned integer array and we use the TextDecoder to convert it into a string. The rest of the logic is the same where we mutate the states to show the response inside the chat bubbles

Demo

The UI invokes the /api/chat api and reads the response from the stream. The connection will be in a pending state until the stream is closed

Conclusion

Streaming the completions from OpenAI is easy to implement. Even though the above demo is based on JS, the same could be implemented with Python or any other language of your choice

Frameworks like langchain also provides callbacks to read from the stream. With streaming, you can avoid the feel of long wait times. You can write your logic to split the responses into separate chat bubbles or perform on-the-fly formatting and a bunch of other things. The above demo project can be found here. Clone it and get creative...

Happy hacking!

References

MDN doc for EventSource

OpenAI libraries for other languages

Langchain streaming

Repository with the demo app

A document Q&A chatbot built with Next.js, supabase and GPT

Neel — Fri, 18 Aug 2023 16:55:58 +0000

I have been trying to build a document Q&A chatbot for the past two months. I initially started off with an application that doesn't rely on GPT, but uses the impira/layoutlm-document-qa model. It didn't work out as the model was not suitable for carrying out causal conversations.

My next attempt involved a python application that extracts the content from the PDF document using tesseract-ocr + pytesseract and passes it to OpenAI api for driving the entire Q&A process. I was just getting started with langchainso there were lot of boiler plate and I was not using it right. That's when I decided to scrap it all and start afresh

Third time is a charm

With the learnings I got from the second try, I stuck to GPT as the core LLM, I chose Next.js 13 as the one-stop solution for building the UI and backend of the application

Before getting into the details, below are the core tools used to build the application

And here is a sneak peek of everything put together

The flow

The user journey of the application and what goes on behind the scenes for each user action are as follows,

⬆️ Uploading the document

The journey starts with the user uploading the PDF document. For this, I used the react-dropzone library. The library supports both drag-and-drop and click-based uploads. Once the user drops the file into the input, the file is sent to the backend for processing

import { useDropzone } from 'react-dropzone';

const { getRootProps, getInputProps, isDragActive } = useDropzone({
    onDrop: (acceptedFiles) => {
        //check if acceptedFiles array is empty and proceed
        //call the API with the file as the payload
        const formData = new FormData();
        formData.append('file', acceptedFiles[0]);
        axios.post("/api/upload", { data: formData });
    },
    multiple: false, //to prevent multi file upload
    accept: {
      'application/pdf': ['.pdf'] //if the file is not a PDF, then the list will be empty
    }
});

The /upload api takes care of three things

Uploading the document to supabase storage bucket
Extracting the content from the document
Persisting the document details in the supabase Database

// Route to handle the upload and document processing

import { NextResponse } from 'next/server';
import { loadQAChain } from 'langchain/chains';
import { Document } from 'langchain/document';
import { PDFLoader } from 'langchain/document_loaders/fs/pdf';
import { supabase } from '../supabase';

export const POST = async (req) => {
  const form = await req.formData();
  const file = form.get('file');

  // Using langchain PDFLoader to extract the content of the document
  const docContent = await new PDFLoader(file, { splitPages: false })
    .load()
    .then((doc) => {
      return doc.map((page) => {
        return page.pageContent
          .replace(/\n/g, ' '); // It is recommended to use the context string with no new lines
      });
    });

  const fileBlob = await file.arrayBuffer();
  const fileBuffer = Buffer.from(fileBlob);

  // Uploading the document to supabase storage bucket
  await supabase()
    .storage.from(bucket)
    .upload(`${checksum}.pdf`, fileBuffer, {
      cacheControl: '3600',
      upsert: true,
      contentType: file.type
    });

  // storing the document details to supabase DB
  await supabase
        .from("documents_table")
        .insert({
            document_content: docContent,
            // insert other relevant document details
        });

  return NextResponse.json({ message: "success" }, { status: 200 });
};

🔃 Initialising Socket.io

The application uses socket.io to send and receive messages in a non-blocking way. After uploading the document successfully, the UI invokes an API - /api/socket to open a socket server connection

Setting up a socket.io server is usually easy, but it was a bit challenging with Next.js 13. The recent versions of Next.js introduced a new paradigm called the AppRouter and setting up a socket server is not possible with it (or at least I couldn't find any documentation for it anywhere). So I had to use the old PageRouter paradigm to initialize a socket server connection

Handler file => src/pages/api/socket.js

Dependencies required => yarn add socket.io socket.io-client

import { Server } from 'socket.io';

export default function handler(req, res) {
  const io = new Server(res.socket.server, {
    path: '/api/socket_io',
    addTrailingSlash: false
  });

  res.socket.server.io = io;

  // When the UI invokes the /api/socket endpoint, it opens a new socket connection
  io.on('connection', (socket) => {
    socket.on('message', async (data) => {
      // For every user message from the UI, this event will be triggered
      const { message } = data;

      // pass on the question and content from the message to Langchain
    });
  });

  res.end();
}

💬 The real chatting

Now that we have the document content handy and the socket open to receive events, it's time to do some chatting.

The UI emits a socket event called message every time the user enters a new message. This message will include 2 important things in the payload,

The actual question
The content extracted from the document (we will get this as the response from the /upload API)

We have already setup an event listener when we initialized the socket server and we will do the actual LLM stuff within this listener to get the answer from GPT.

Dependencies required => yarn add openai pdf-parse langchain

import { Document } from 'langchain/document';
import { loadQAStuffChain } from 'langchain/chains';
import { llm } from '@/app/api/openai';

export default function handler(req, res) {
  const io = new Server(res.socket.server, {
    path: '/api/socket_io',
    addTrailingSlash: false
  });

  res.socket.server.io = io;

  // When the UI invokes the /api/socket endpoint, it opens a new socket connection
  io.on('connection', (socket) => {
    socket.on('message', async (data) => {
      // For every user message from the UI, this event will be triggered
      const { question, content } = data;

      const llm = new OpenAI({
          openAIApiKey: process.env.OPENAI_API_KEY,
          modelName: 'gpt-3.5-turbo'
      });

      // We will be using the stuff QA chain
      // This is a very simple chain that sets the entire doc content as the context
      // Will be suitable for smaller documents
      const chain = loadQAStuffChain(llm, { verbose: true });

      const docs = [new Document({ pageContent: content })];

      const { text } = await chain.call({
        input_documents: docs,
        question
      });

      // Emitting the response from GPT back to the UI
      socket.emit('ai_message', { message: text })
    });
  });

  res.end();
}

In the above snippet, we use the Stuff QA chain which is a simple and suitable one for smaller documents. We generate a new Document object array with the content of our target document. The final step is invoking the call method to pass on the payload to OpenAI api and get the response

Behind the scenes, langchain generates a prompt in the following format and sends it to the OpenAI api to get the answer

Use the following pieces of context to answer the question at the end. If you don't know the answer, just say that you don't know, don't try to make up an answer.

<the_entire_document_content>

Question: <some question>?

Helpful Answer:

The above prompt will be the same for every question you ask and the document content will be passed over as the context.

On receiving the response from the API, we emit a new event called ai_message. The UI has an event listener for this event and we handle the logic of displaying the chat bubbles based on the message

socket?.on("ai_message", async (data) => {
  setConversations((prev) => {
    return [
      ...prev,
      {
        user: "ai",
        message: data.message,
      },
    ];
  });
});

This concludes the entire flow

⚙ Behind The Scenes

The code snippets mentioned above are the trimmed version with only the key items required for this article. Below are some of the things that the application handles along with the document QA

Persisting the document details: The document details such as the checksum of the document, original document name and the content of the document are stored in the supabase DB. This data will be used to display a chat history on the UI and enables the user to return to their conversations anytime

Persisting the conversations: The chat messages sent by the user and generated by the AI are also persisted in the database. The user can click on any document's chat history and see all the exchanged conversations

Storing the actual document: The original PDF document uploaded by the user is stored in supabase storage bucket. This is for enabling the user to download the document from the chat section to see the originally uploaded content

User authentication: I wanted to try supabase authentication, so I added a login flow to the application. With supabase's Row Level Security (RLS) and policies, the chat history and the conversations will be shown only to the authenticated users

🙋🏻 One more thing...

If your aim is just to chat with a small document with just a couple of pages, then you can skip this section

Until this point whatever we have seen will do good for documents with no more than 4 pages of textual content. For instance, if you want to chat with a research paper that often goes over 10 pages then it becomes a scalability issue to send the entire content of the document back and forth for every question. I initially tested it out with the well-known research paper about transformers that has over 40k characters and the application choked out with that content. So I had to rethink the solution

Embeddings to the rescue... Embeddings are vectors that hold a bunch of numbers that denote the similarity or closeness between different sentences or words. To tackle the problem, I did the following

Split the extracted document content into smaller chunks

Generate OpenAI embeddings for each chunk

import { PDFLoader } from 'langchain/document_loaders/fs/pdf';
import { CharacterTextSplitter } from 'langchain/text_splitter';
import { MemoryVectorStore } from 'langchain/vectorstores/memory';

export const extractDocumentContent = async (file) => {
  const chunks = await new PDFLoader(file).loadAndSplit(
    new CharacterTextSplitter({
      chunkSize: 5000,
      chunkOverlap: 100,
      separator: ' '
    })
  );

  const openAIEmbedding = new OpenAIEmbeddings({
    openAIApiKey: process.env.OPENAI_API_KEY,
    modelName: 'text-embedding-ada-002'
  });

  const store = await MemoryVectorStore.fromDocuments(chunks, openAIEmbedding);

  const splitDocs = chunks.map((doc) => {
    return doc.pageContent.replace(/\n/g, ' ');
  });

  return {
    wholeContent: splitDocs.join(''),
    chunks: {
      content: splitDocs,
      embeddings: await store.embeddings.embedDocuments(splitDocs)
    }
  };
};

Store the chunk and its respective embeddings in subapabse DB. This blog can be referred to know how to work with vector data types in supabase

const saveDocumentChunks = async (file) => {
  // Invoke the function from above to get the chunks
  const { chunks } = await extractDocumentContent(file);
  const { content, embeddings } = chunks;

  // store the content from the chunk and its respective embedding to the DB
  // for context, a simple embedding vector will look something like this
  // [-0.021596793,0.0027229148,0.019078722,-0.019771526, ...]
  for (let i = 0; i < content.length; i++) {
    const { error } = await supabase
      .from('document_chunks')
      .insert({
        chunk_number: i + 1,
        chunk_content: content[i],
        chunk_embedding: embeddings[i] // chunk_embedding is of type `vector` in the Database
      });

    if (error) {
      return { error }
  }

  // Be mindful of implementing a rollback strategy even if storing a single chunk fails
  return { error: null };
};

Do a similarity search on the vector Database using the user's question to filter only the relevant chunks (We need to set up a supabase plpgsql function to rank the chunks based on similarity)

create function match_documents (
    query_embedding vector(1536),
    match_count int default null,
    filter_checksum varchar DEFAULT ''
) returns table (
    document_checksum varchar,
    chunk_content text,
    similarity float
) language plpgsql as $ $ #variable_conflict use_column
begin return query
select
    document_checksum,
    chunk_content,
    1 - (
        document_chunks.chunk_embedding <= > query_embedding
    ) as similarity
from
    document_chunks
where
    document_checksum = filter_checksum
order by
    document_chunks.chunk_embedding <= > query_embedding
limit
    match_count;
end;
$ $;

Use the filtered chunks as the context for answering the questions

import { Document } from 'langchain/document';
import { loadQAStuffChain } from 'langchain/chains';
import { OpenAIEmbeddings } from 'langchain/embeddings/openai';

const inference = async (question) => {
  const openAIEmbedding = new OpenAIEmbeddings({
    openAIApiKey: process.env.OPENAI_API_KEY,
    modelName: 'text-embedding-ada-002'
  });

  const { data, error } = await supabase.rpc('match_documents', {
    query_embedding: await openAIEmbedding.embedQuery(question),
    match_count: 5,
    filter_checksum: "unique_document_checksum"
  });

  if (error) return { error };

  const content = data.map((v) => v.chunk_content}).join(' ')

  // set the `content` as the context of the prompt and ask the questions
  const chain = loadQAStuffChain(llm, { verbose: true });
  const docs = [new Document({ pageContent: content })];

  const { text } = await chain.call({
    input_documents: docs,
    question
  });

  return { answer: text };
};

With these improvements in place, I uploaded the same research paper and started a conversation. To my surprise, it worked on the first try and gave back the answers with ease

In this approach, you need not pass the document content for every message because the content will be fetched from the DB based on the question and its relevance to the content

🚀 Where is the code?

I have published the entire project on Github with the instructions to set up the project locally and to set up the supabase project

Github project

🏁 Conclusion

Like they say, "third time is a charm". Finally, after two failed attempts, I have built an application that works fine with all the integrations and the accuracy with which GPT handles the questions feels mystical sometimes. You can try out the application locally by cloning the repository and running it. Just ensure that you have set up a working supabase project and are ready to spend a few bucks on OpenAI.

Happy hacking!

Further reads

Stream the response from GPT

📚 References

Langchain

Next.JS 13

Supabase

https://supabase.com/blog/openai-embeddings-postgres-vector

https://js.langchain.com/docs/modules/data_connection/vectorstores/integrations/supabase

https://js.langchain.com/docs/api/document_loaders_fs_pdf/classes/PDFLoader

https://js.langchain.com/docs/api/text_splitter/classes/CharacterTextSplitter

Generate an image grid using Python

Neel — Mon, 10 Jul 2023 18:16:24 +0000

I have been working with some image processing stuff recently and I am surprised to see that the Python ecosystem has a plethora of libraries available to solve every image processing use case

In my previous post, I covered how to convert a PDF document to a set of images. This post is somewhat the reverse of it, but not entirely. We will see how to generate an image grid by combining a set of images using Python and PyTorch Vision

All you need for this is torchvision installed on your machine and a few images lying around to be trapped into a grid

pip install torchvision

These are the individual images that will be added to the resultant grid

I like Python for its simplicity because oftentimes, complicated problems could be ground down to just a few lines of code. Below is all the code it takes to generate an image grid

from torchvision.io import read_image
from torchvision.utils import make_grid, save_image
from torchvision.transforms import transforms
from torchvision import torch

import os

tensors = []

transform = transforms.Compose([
    transforms.CenterCrop(1000),
    transforms.ConvertImageDtype(dtype=torch.float),
])

for file in os.listdir('./images'):
    image = os.path.join('./images', file)
    transformed_tensor = transform(read_image(image))
    tensors.append(transformed_tensor)

grid = make_grid(tensors, nrow=2, padding=5)

save_image(grid, "grid.jpg")

Now let's see what each block does

Transformers

transform = transforms.Compose([
    transforms.CenterCrop(1000),
    transforms.ConvertImageDtype(dtype=torch.float),
])

The above block includes a list of transformers to transform the images into the desired vector.

torchvision expects all the input images to be of the same dimension. If the dimension of your first image is [1024 x 480] then the rest of the images should be of the same dimension, else you will be greeted with an exception

RuntimeError: stack expects each tensor to be equal size, but got [3, 816, 1456] at entry 0 and [3, 1024, 1024] at entry 1

To mitigate this, we are cropping the image from the center using CenterCrop with a uniform dimension (square dimension) of [1000 x 1000]. If the original image dimension is smaller than this value, then the empty space will be filled with black pixels (like the back letterbox bars you see in a movie to preserve the aspect ratio)

The crop transformer returns a tensor with uint8 as the datatype, but the save_image function expects the dtype to be float. This is when the ConvertImageDtype transformer comes in handy, using which we can convert the tensor datatype to float

If you have a list of images with the same dimension then feel free to skip the transformers

Read the images

for file in os.listdir('./images'):
    image = os.path.join('./images', file)
    transformed_tensor = transform(read_image(image))
    images.append(transformed_tensor)

Once the transformers are setup we can directly go ahead and read the images. The make_grid function expects a list of tensors and the following are the steps required to achieve just that

List the directory to get all the image file names
Use read_image function from the torchvision.io module to read the images one by one
The read_image function returns a tensor and we transform the tensor to the desired form using the transformers, which we have already setup
Store the tensors in a List that will be used to generate the grid

Generate and save the grid

grid = make_grid(images, nrow=2, padding=5)
save_image(grid, "grid.jpg")

The make_grid function expects the tensor list to be the first argument.

nrow argument defines the number of images you expect to see in a single row
padding argument just adds padding between the images in the grid

You get a tensor back from the make_grid with float as the datatype and the same can be passed to the save_image function to save the grid as an image file (grid.jpg in this example)

The following is the result

Conclusion

If you are not familiar with PyTorch, then it is a widely used deep learning library. The torchvision is a part of the PyTorch project and it is used for solving complex computer vision problems. This post covers a very minuscule problem that I recently solved in one of my projects

To learn more about this library, you can take a look at the official docs

PyTorch

Torchvision

Google colab notebook

Convert a PDF document into images using Python

Neel — Thu, 29 Jun 2023 10:00:36 +0000

I was working on a Python project recently (for a DEV hackathon, of course) and I needed a library to extract the pages from a huge PDF document as images. The primary use case was to extract the content from the PDF document and doing this in a blocking way was expensive. So I was looking for options to split the document's pages into multiple images which would let me concurrently do the content extraction.

As any rad developer would do, I asked ChatGPT for a solution and got a pretty easy-to-integrate option to achieve my goal.

The solution is to use Python pdf2image package and write just 2 lines of code to get the pages from the PDF document extracted as images.

If you are not interested in the specifics of the process and get directly into the code, then you can jump straight to the pdf2image notebook

Setup

Even though the solution is pretty simple, you need to set up a utility called as poppler before you start writing your Python code. Poppler is the underlying C++ based PDF rendering library used by pdf2image for rendering the PDF document behind the scenes.

MacOS

Setting up poppler on a MacOS device is as simple as running brew install poppler (assuming you have Homebrew installed on your machine)

Linux

If you are on a Linux machine, which is very common if you are running your application as a container, then the process of installing poppler may differ based on the distro you are opting for

If its a Ubuntu based machine/container, then you can install poppler using apt-get -> sudo apt-get install poppler-utils

Windows

The process is slightly elaborate on Windows (just like everything else on Windows 😋). The latest releases for the Windows binary can be found here.

Extract the zip file and add the <extracted folder>/Library/bin to the PATH environment variable

Time for some code

Once the setup is done you can install the pdf2image package from pip and get on with your coding

pip install pdf2image

Convert from path

If the PDF file you want to convert is nicely located in a directory, then the job is much easier. You can directly import the convert_from_path function from the pdf2image package to extract the pages as images

from pdf2image import convert_from_path
import os

# Checks if the output folder exists and if not it creates a new one
if not os.path.exists("extracted_images"):
    os.mkdir("extracted_images")

convert_from_path("top_secret_document.pdf",
                  output_folder="extracted_images", fmt="jpeg")

The above code will pick the PDF document named top_secret_document.pdf from the current path and extracts every single page in the document as a JPEG image and stores it in the output folder named extracted_images.

The function expects the output folder to be already available and if it is not available, then an exception will be raised. To avoid this, you can conditionally check for the existence of the folder and create it, if not present

The good thing about this function is that it assigns a UUID to the output images and appends a page counter to the file name to make the file names unique. If you want to add a common prefix instead of going with a UUID, then you can make use of the output_file parameter accepted by the function

Convert from bytes

In the project I was working on, the use case goes like this.

The users are asked to upload the document and the uploaded file is submitted to a Golang based microservice. This service does some validations and other stuff behind the scenes. Once all the processing is done, it stores the document in a minio bucket and publishes an event to a Kafka topic

The python service acts as a consumer and once the event is received, it fetches the PDF document from the bucket and starts with the page extraction process. This means that the document will not be readily available in the path of the file system and it will be available as a byte stream.

This is not an esoteric topic to the pdf2image library and it readily comes with a convert_from_bytes function to tackle such use cases

# The get_object function fetches the PDF document stored in the object store
response = minio_client.get_object(bucket_name, "top_secret_document.pdf")

document_as_bytes = response.read()

images = convert_from_bytes(document_as_bytes, fmt="jpeg", output_folder="extracted_images")

The above code will read the PDF document as bytes and store the pages into image files in the output directory.

This function is not limited to PDF documents downloaded from external sources. You can use it to read a file from the file system as bytes and use that to extract the images

from pdf2image import convert_from_bytes

f = open('./top_secret_document.pdf', 'rb') #opening the file as a binary file

document_bytes = f.read()

images = convert_from_bytes(
    document_bytes, fmt="jpeg", output_folder="extracted_images")

Provided we already have the convert_from_path function to abstract all this boilerplate, I highly doubt if someone is ever gonna use the convert_from_bytes function for working with a local PDF file, still I just wanted to show that it's possible

Conclusion

That's all to it. The pdf2image library is not limited to light PDF documents and it can work with bulk documents too. Both the convert_from_path and convert_from_bytes variety of arguments that can be used to control the process of rendering the PDF document and extracting the images in different ways

I have attached some useful links in the reference section below which can be used to explore the pdf2image library in depth

The code used above can be found in the following Collab notebook pdf2image notebook

🐍!Happy coding!🐍

References

Official library docs

Poppler

Utility to mock REST endpoints in real-time

Neel — Fri, 29 Jul 2022 15:05:00 +0000

Failure is success in progress - Albert Einstein

Prelude

The path to success is never meant to be easy and the destination is never meant to be the success itself. It's always about the journey.

In this article, I would like to introduce everyone to one of the peculiar open source projects that we have been working on for the past few months. We are a team of 2 young engineers [Neel and Aravind], who were keen to learn right from the start of our career. The passion for learning and building projects for the community is one of the burning desires we both have in common.

Mimock - Mock REST endpoints in real-time

This is a great utility tool that helps us to set up mocks for REST API endpoints to test UI applications locally or in CI/CD pipelines. Mimock is an interceptor-based mocking solution that enables the developer to create or update mocks without having to restart the application.

Story of why we built mimock

In the current technological world, the way in which an application is conceptualized, developed, deployed, and brought forward to market is a mind-blowing process. These different applications, projects, and tools built by startups, MSMEs, or even big enterprise companies are hugely reliant on the open source community.

The current trend towards open source contributions is heavily growing. Awesome engineers around the world have started contributing more and more to multiple different projects and even big tech companies now are playing a greater role for the good of the open source community. We were also planning to develop one such open source tool which should be useful as well as something that is easier to use. We tried to identify some of the problems which the developers were facing and tried to figure out solutions for the same.

Problem Statement:

What if the developer or QA can mock a REST API endpoint to mimic its response, without writing any implementation code for the mock itself?

This was the initial step toward conceptualizing our peculiar utility tool.

Mimock is developed for the purpose of setting up mocks for REST endpoints without the hassle of maintaining any code for the mocks. The application exposes a React-based UI which can be used by anyone with no prior coding experience to create and maintain mocks.

If a UI application is built with a configurable origin for the backend it is consuming, then mimock can be wired up in its place to serve the required response. This is applicable for backend services as well which rely on other APIs or services to process a request.

Some examples use cases are as follows,

If your UI (Web or Mobile) relies on an external metered API that incurs cost based on the total hits, then mimock can be used in its place to mimic that API and serve the response on lower environments or CI/CD pipelines that runs UI automation tests
If a backend API relies on another endpoint to download a PDF document and if that PDF server's latency is high which is not bearable for just testing the service locally or while running integration tests on pipelines, then mimock will come in handy for serving the required PDF document

Features

Unique features offered by the Mimock utility tool are as follows,

Intuitive UI: Provides an intuitive UI that lets anyone manage mocks without any coding experience
No Re-Deployment: Mocks can be added in real-time and no application restart is required. Mocks are created/updated on the go which ensures faster development and turn-around time
Access Management: The platform follows a role-based user model and the admin can assign roles to the users to set up and restrict the access
Text and Binary Response: With the support for both text and binary response types, setup your mock to serve a normal JSON response or a JPEG image file or even your desired PDF document

How we achieved to setup mock

We created a spring boot web application that hosts the logic to manage and store the mock REST API endpoints. This web application also performs the pattern matching for the stored mocks whenever they are hit by the respective client/UI application which is accessing the targeted REST API endpoint.

The below section explains one good use case of our platform,

Let us consider an application that has a separate backend and frontend hosted internally. The backend service exposes the APIs and the frontend application provides the user control.

The flow of the request and response diagrams before and after mimock are shown and explained as follows,

Actual interaction without mimock:

The UI application usually sends out the request to the REST API endpoint exposed by the backend service. Based on the API request sent out by the client the backend service performs the business logic and sends out the relevant response back to the UI.

In this case, the request-response flow depends upon the request headers, request body, query parameters, and so on and behaves differently for each and every API. This is the traditional setup we have seen so far in most of the projects.

The cost involved for the UI team to wait out the implementation of the API by the backend team is highly time-consuming. Let alone the cost involved when the API has to be re-deployed whenever there is a new change or feature enhancement for the API. If let’s say the API belongs to some external team the requested feature may or may not be provided in the stipulated timeline and the rate-limiting will be one major factor if it is a paid API.

The UI team also has the testing phase which involves hitting endpoints and analyzing the behavior of the UI using automated tests such as selenium or puppeteer. These tests run mostly in the CI/CD pipeline and the reliance on the backend service is very heavy.

Mimock as a platform acts as a replacement for these backend services deployed along with the UI application dev or testing environments. However the deployment behavior for integration environments, where both the real frontend and backend services, still stay the same.

In the project ideation phase, the best practice is to actually plan out the behavior of the API way ahead of its implementation. Most of the teams decide upon the API contract which will include the request and response details. This is a great advantage to us and why not use it in an automated smart way via a tool.

After introducing mimock:

Mimock platform acts as a replacement for the whole backend service. Here the respective client/UI application sends out the request and the mimock platform intercepts the request and responds based on the dynamic parameters such as query params, request body, and request headers.

Step by Step explanation:

The team uses the mimock platform to store the mocked REST API endpoints. (either via REST APIs exposed by us or the UI)
They are allowed to store multiple mock behaviors which they might expect and the relevant request and response details associated with them.
Now whenever the UI application sends out a request, the mimock platform intercepts the request and matches the route based upon the mocked API stored in its database and responds with the relevant response.

In this way, the mimock platform totally eliminates the reliance on the backend service for the REST API endpoints.

There is also an added bonus here as there is no re-deployment required since the mocks can be changed in real-time and the platform responds to the updated mocks with a faster turnaround time.

Automated UI journey tests can be run in CI/CD pipelines with the help of mimock by either running an ephemeral instance within the pipeline or by hosting mimock as an external service and accessing the mocked endpoints

Tech Stack

Our tech stack involves,

Spring Boot - For the core backend
React - For the intuitive UI
Postgres - For storing the mocks and user information
Docker - For distribution

Items we have in our roadmap

Some of the items we have in our roadmap includes as follows,

Import Mocks in Bulk
Regex matching for dynamic fields in query params, request body, and request headers
Dynamic text response generation using the fields from the request (query param, request body, or request headers)
Cloning mocks
Grouping mocks to folders/categories
File Preview for binary response mocks in the mimock platform

and much more on the way!!!

Reach out to us

You can reach out to us via the below channels,

Remote debugging Github actions workflow

Neel — Fri, 08 Apr 2022 17:19:25 +0000

I was in the process of setting up a new deployment workflow for one of my GitHub repositories. I tested all the scripts locally that I added to the GitHub actions workflow file (.github/workflow/deployment.yml) and to my relief all the scripts worked as expected. On that positive note, I pushed the new workflow file and was waiting eagerly to see the little yellow dots on Github actions turn green

Well, it did turn green but I realized that something was not right after validating the deployed artifacts. (If things go as expected in the first try then our jobs as devs wouldn't be that interesting now, would it ?)

The issue was that the workflow missed to include some required assets in the deployed bundle. The same script which worked locally without any issues was misbehaving in the pipeline, and I spend an hour scrutinizing the yaml file line by line and I could see nothing deviating from what the workflow was intended to do. So I concluded that somehow my setup is not compatible with the worker that GitHub uses to run my workflow.

I was looking for an option to directly get into the worker that is running my workflow so that it would be easier to troubleshoot the bug. I was on the looking spree for a tool that does the same and after a very short time of research, I found the perfectly suitable one

Action: https://github.com/csexton/debugger-action

TL;DR The above action can be included in the workflow yaml file and it will keep the job hanging from the stage where the action was referred. It will print the SSH details to the console which can be used to remotely login to the GitHub worker that is running the workflow

How to set it up?

Setting up this action is very simple. You just need to add the following code to your workflow yaml file and this action will block your job from that stage onwards

- name: Setup Debug Session
  uses: csexton/debugger-action@master

Here is a sample workflow file with the complete setup

name: New Pipeline

on:
  push:
    branches: [ main ]

  pull_request:
    branches: [ main ]

jobs:
  debug-workflow:
    name: Job to debug the workflow
    runs-on: ubuntu-latest
    steps:
      - name: Check out code
        uses: actions/checkout@v2

      - name: Setup Debug Session
        uses: csexton/debugger-action@master

Once the job is blocked, if you expand the details of that stage you will get the details to login remotely to the worker using SSH

If you have a working terminal capable of logging into a remote host using SSH, then just running ssh xxxxxxxxxxxx@sfo2.tmate.io should directly take you to the worker. The default session timeout is set to 15 minutes and if you wish to extend it, then running the command touch /tmp/keepalive should do the trick.

Remote session of the github actions worker

From the SSH session, if you know your way around remote sessions and the commands for your host then you can troubleshoot your issues with ease.

In my case, the issue was due to an improperly configured path relative to the working directory which I found out once I tried to run the commands manually within the worker.

Conclusion

This is such a simple-to-setup action that comes in handy when you are setting up new GitHub actions workflows and if you wish to ensure that the internals are being setup properly (Let's not leave room for any surprises on the day of deployment)

Which one is the right way to maintain the UI and back-end code for an open-source project?

Neel — Mon, 25 Jan 2021 14:56:03 +0000

I have been working on an open-source project for almost a year now and since the beginning I have been maintaining the front-end and the back-end codebase as two separate repos on github.

This is because the front-end is based on React JS and the server code is based on golang. So I assume that if I coupled both into a single repo, then it could overwhelm the contributors. I have a github action which binds the UI and back-end during the build stage, and pushes each commit to a unified repo, which can be used for building the application from source.

I have taken a look at a few similar projects on github and all have a single unified repo, with both the front-end and back-end under a single tree.

Is my 2-repo approach right or do I need to maintain it as a single repo with a clean documentation to guide the contributors. Please share your thoughts.

A project status tracker crafted with Next JS

Neel — Sat, 09 Jan 2021 16:49:57 +0000

A tracker built with Next JS for tracking the status of your projects

Category Submission: Random Roulette

https://aplaza-kezvw.ondigitalocean.app/

Screenshots

Description

Aplaza is a mini web based tracker for keeping track of all your projects stored in a single data store.

Link to Source Code

https://github.com/neel1996/aplaza

Permissive License

MIT

Background

(What made you decide to build this particular app? What inspired you?)

I always wanted to learn Next JS and this hackathon gave me the opportunity to do so. I am a huge fan of Typescript and with this project, I had a chance to explore TS further.

The project utilizes either a JSON data-store or a configured PostgreSQL Database. This can be configured under a TS file in the api/data directory.

How I built it

(How did you utilize DigitalOcean’s App Platform? Did you learn something new along the way? Pick up a new skill?)

The project was built with SOLID principles in mind and it was developed in such a way that it accepts changes swiftly without impacting dependent modules.

For this hackathon, I am submitting the initial release (v0.0.1) of the project but I am planning to expand this project further with additional features.