Forem: Leo Antony

CLIP-Powered Multi-Modal Search with Redis Vector Index and Graph

Leo Antony — Sat, 09 Aug 2025 17:11:16 +0000

This is a submission for the Redis AI Challenge: Real-Time AI Innovators.

What I Built

I built a real-time, multi‑modal semantic search system that combines vector similarity search with graph‑based context expansion. Text and images are embedded locally using CLIP (openai/clip-vit-base-patch16, 512‑dim). Redis 8 powers the vector index (cosine distance) and fast KNN lookups, and also serves as a cache for query results. On top of the nearest neighbors, I construct and traverse a semantic graph using NetworkX in Python to discover related items beyond the initial top‑K—enabling richer, more explainable retrieval.

Key features:

Local CLIP embeddings for text and images
Redis 8 vector index with cosine similarity + KNN search
Result caching in Redis for low latency and reduced recompute
NetworkX semantic graph to link and rank related items across searches
Duplicate prevention via SHA‑256 content hashing
Endpoints for submit (ingest) and search.

Demo

Screenshots:

How to try:

Submit text or an image.
The system embeds the input, stores/updates it, and runs a KNN search.
Results are cached in Redis; a NetworkX graph expands context to suggest related items.

How I Used Redis 8

I used Redis 8 as the real-time data layer for indexing, search, and caching:

Vector Index (RediSearch): I created a VECTOR HNSW index on a hash field named embedding (DIM=512, DISTANCE_METRIC=COSINE). All embeddings are stored as raw float32 bytes. KNN queries use FT.SEARCH with PARAMS to pass the query vector and SORTBY vector_score.
Semantic Caching: I cache search responses keyed by a normalized query hash (e.g., sha256 of lowercased text). Hot queries return instantly from Redis without re‑embedding or re‑searching, with a short TTL to ensure freshness.
Metadata and Documents: Each item is stored as a Redis hash with fields like embedding (bytes), type (text/image), content or URL, and submit_count. This allows quick retrieval for UI assembly.
Deduplication: Before storing, I normalize and hash the input content. If the key exists, I update usage stats instead of duplicating vectors or nodes.
Performance: Redis delivers sub‑millisecond lookups on cached results and fast vector KNN even at scale thanks to HNSW. Caching also reduces load on the local embedding server and keeps latency predictable.

solo Project by: @leoantony72

leoantony72 / multi_model_vectorSearch

Multi Model vector search with Redis

Real-Time Multi-Modal Semantic Search System

This project is a real-time, multi-modal semantic search system that combines vector similarity search with graph-based context expansion.
It supports both text and image search using locally computed CLIP embeddings.

📜 What I Built

Local CLIP embeddings for text and images (openai/clip-vit-base-patch16, 512-dim)
Redis 8 vector index with cosine similarity and KNN search
Result caching in Redis for low latency and reduced recompute
NetworkX semantic graph to link and rank related items beyond the initial top-K → enables richer, more explainable retrieval
Duplicate prevention using SHA-256 content hashing
Endpoints for
- submit (ingest)
- search (retrieve)

🛠 Architecture Overview

Embed: Text and images embedded locally via CLIP
Index: Vectors stored in Redis 8 (cosine similarity)
Search: Fast KNN lookups in Redis
Expand: Related items discovered via semantic graph traversal in NetworkX
Cache: Query results cached in Redis
Serve…

View on GitHub

Suggestions

As the development of AI increases, getting relations faster is a major step, While I was sad knowing that redis stopped support for redis graph. I encourage the redis Team to bring back support for Graph.

File Organizer Desktop app with Go🚀

Leo Antony — Mon, 19 Aug 2024 00:18:08 +0000

While Go is famous for building APIs and CLI tools, but it's potential for building desktop apps is less commonly discussed😑. Popular languages such as JavaScript/Rust already provide packages to built desktop apps such as Electron.js for JS and Tauri for Rust.

However Electron.js is criticized consuming lot of memory leading many companies to switch to alternatives like Tauri. Hoppscotch recently released their desktop app with Tauri, highlighting the growing trend toward more resource-efficient solutions.

Go is also leading the way in building Desktop apps with wails, and with wails and Vue.js I built a File Organizer which sort the files to their respective folders based on the File Type, Year/Month of creation.

Serpico: File Organizing App🚀

Simplify your digital life with Serpico, the ultimate app for organizing your files with precision and ease. Designed to help you manage and declutter your digital space, Serpico organizes your files by month, year, and file type.

Key Features

Automatic File Organization: Serpico intelligently categorizes your files by month, year, and file type, making it easy to navigate through your digital archive.
User-Friendly Interface: Enjoy a clean and intuitive interface that makes organizing and accessing your files a breeze🐱‍💻.
Secure and Reliable: Trust Serpico to handle your files securely, with robust encryption and backup options to keep your data safe.
With Serpico, managing your digital files has never been easier. Whether you're a professional, a student, or just looking to declutter your personal files, Serpico is your go-to solution for a more organized digital life.

Screenshots

Development

To run in live development mode, run wails dev in the project directory. This will run a *Vite * development server that will provide very fast hot reload of your frontend changes. If you want to develop in a browser and have access to your Go methods, there is also a dev server that runs on http://localhost:34115. Connect to this in your browser, and you can call your Go code from devtools.

To build a redistributable, production mode package, use wails build.

Conclusion

I encourage you all to try out wails, It's easy to use and they a good documentation if you want that, wails.

Email: leoantony102@gmail.com
GitHub: https://github.com/leoantony72

Repo: https://github.com/leoantony72/serpico
Download Link: https://github.com/leoantony72/serpico/releases/tag/v1.0

Go High-Performance Cache with TTL and Disk Persistence

Leo Antony — Tue, 13 Aug 2024 15:15:03 +0000

1.Introduction

Accelerate your Golang projects without the hassle of setting up a database everytime you begin a new project. Tired of configuring databases from scratch? Only to face new problems? Look no further In this blog we will be looking into Golang caching library with Support for TTL, disk Persistence and Hash Data type.

GoSwift.

2.Prerequisite

Basic Knowledge of Golang
Knowledge of how a typical cache works

3.Features

Set & Get command
Update command
Del command
Exists command
Support for TTL
Support for Disk Save(Snapshots)
Support Hash Data type (Hset, Hget, HgetAll, HMset)
Safe Locking

Set & Get command


import (
    "fmt"
    "github.com/leoantony72/goswift"
)

func main(){
    cache := goswift.NewCache()

    // Value 0 indicates no expiry
    cache.Set("key", "value", 0)

    val, err := cache.Get("key")
    if err !=nil{
        fmt.Println(err)
        return
    }
    fmt.Println("key", val)
}

Update command

// Update value
// @Update(key string, val interface{}) error
err = cache.Update("key","value2")
if err != nil{
    fmt.Println(err)
    return
}

Del command && Exists command

// Delete command
// @Del(key string)
cache.Del("key")

// Exist command
// @Exists(key string) bool
value = cache.Exists("key")
fmt.Println(value) // returns false

Support for TTL

// Set Value with Expiry
// @Set(key string, val interface{}, exp int)
// Here expiry is set to 1sec
cache.Set("key","value",1000)

// Hset command
// @Hset(key, field string, value interface{}, exp int)
// in this case the "key" expires in 1sec
cache.Hset("key","name","value",1000)
cache.Hset("key","age",18,1000)

Support Hash Data type (Hset, Hget, HgetAll, HMset)


// Hset command
// @Hset(key, field string, value interface{}, exp int)
// in this case the "key" expires in 1sec
cache.Hset("key","name","value",1000)
cache.Hset("key","age",18,1000)


// HMset command
// @HMset(key string, d interface{}, exp int) error
// Set a Hash by passing a Struct/Map
// ---by passing a struct---
type Person struct{
    Name  string
    Age   int
    Place string
}

person1 := &Person{Name:"bob",Age:18,Place:"NYC"}
err = cache.HMset("key",person1)
if err != nil{
    fmt.Println(err)
    return
}

// ---by passing a map---
person2 := map[string]interface{Name:"john",Age:18,Place:"NYC"}
err = cache.HMset("key",person2)
if err != nil{
    fmt.Println(err)
    return
}


// Hget command
// @HGet(key, field string) (interface{}, error)
// get individual fields in Hash
data,err := cache.HGet("key","field")
if err != nil{
    fmt.Println(err)
    return
}
fmt.Println(data)

// HgetAll command
// @HGetAll(key string) (map[string]interface{}, error)
// gets all the fields with value in a hash key
// retuns a map[string]interface{}
data,err = cache.HGetAll("key")
if err != nil{
    fmt.Println(err)
    return
}

Snapshots

opt := goswift.CacheOptions{
        EnableSnapshots:  true,
        SnapshotInterval: time.Second*5,
    }
c := goswift.NewCache(opt)

This will take a snapshot of the Data Every 5sec and saves it into a Snapshot.data file. By default Snapshots are disabled and if the SnapshotInterval is not provided default value is 5seconds.

NOTE: If the EnableSnapshot is false, Data saved in the file will not imported

Error handling

const (
    ErrKeyNotFound   = "key does not Exists"
    ErrFieldNotFound = "field does not Exists"
    ErrNotHashvalue  = "not a Hash value/table"
    ErrHmsetDataType = "invalid data type, Expected Struct/Map"
)

These are the common Errors that may occur while writing the code. These Varible provide you a clear and easy Error comparison method to determine errors.

data,err := cache.Get("key")
if err != nil {
    if err.Error() == goswift.ErrKeyNotFound {
        //do something
}
}

Inner workings of the cache expiry

Every 3sec the **sweaper **function gets called to clear out the expired values from the hash table. We maintain a min heap which points to the hash map. The top element will be the key with the smallest TTL. we traverse through the tree until TTL is greater then the current time.

Summary

I wouldn't advice you to use this in production!!, but feel free to use it in your small side project. Do try it out and if you encounter a bug do make a issue on GitHub repo.

Email: leoantony102@gmail.com
Github: https://github.com/leoantony72
Repo: https://github.com/leoantony72/goswift

How I Made Twitter Back-end

Leo Antony — Sat, 15 Apr 2023 07:35:27 +0000

1. Introduction

I am designing a back-end for Twitter using a micro-service architecture. This design is based on the implementation by Twitter's engineers, and I had made some slight modifications to enhance scalability and improve performance. I will be coding and implementing this design as I progress.

GitHub Project Link: Twitter Backend

2.Prerequisite

Basic system Design concepts like:

Jwt
Caching
load balancer
Message Queue/Bus
Service Registry

3.Features

User signup/Login - JWT
User should be able to tweet.
User can Delete Tweet
User can see other user's tweet timeline(with retweets).
User can get their Home timeline.
User can follow/unfollow others.
System should be Scalable.

4.Challenges we face

Twitters high volume of daily tweets can be challenging for scaling the platform. Twitter by nature is a Read Heavy application compared to writes. meaning there are far more users consuming tweets than users actively tweeting. Twitter still produce 5800 tweets per/second btw😬.
Since being a Read Heavy application the database will experience heavy load. We can reduce the heavy load by using a cache like Redis.
When a tweet is tweeted it is send to queue so it might take a bit to show up on user's Home Page, It's okay but we have to make sure not to increase this time gap.
Jwt key sharing among other service for authentication

5.Database Design

In this Users table have One to Many relations with Tweets,Likes,Retweets,User_follows.

6.Implementation

Auth Service

Using session would require to make a call to the database every time a request comes in. Since we are following microservice architecture we will be using JWT to reduce database calls.

For Verifying these Jwt token we would have to share the master key to every service which is not the best practice. What's the solutions Asymmetric Key!. Basically Asymmetric key means that we will Encrypt with a key and Decrypt with another key.

The master key will be used to make the tokens and slave key can only be used for verifying these tokens. For Creating Asymmetric token we need to generate a RSA key then we create a private & public certificate.



// generate key
privatekey, err := rsa.GenerateKey(rand.Reader, 2048)
if err != nil {
    fmt.Printf("Cannot generate RSA key\n")
    os.Exit(1)
}
publickey := &privatekey.PublicKey

Then we dump this key to private.pem file for further use in generating jwt tokens



// dump private key to file
    var privateKeyBytes []byte = x509.MarshalPKCS1PrivateKey(privatekey)
    privateKeyBlock := &pem.Block{
        Type:  "RSA PRIVATE KEY",
        Bytes: privateKeyBytes,
    }
    privatePem, err := os.Create("private.pem")
    if err != nil {
        fmt.Printf("error when create private.pem: %s \n", err)
        os.Exit(1)
    }

do this for the public.pem certificate, we use this certificate to verify jet token.



// dump public key to file
    publicKeyBytes, err := x509.MarshalPKIXPublicKey(publickey)
    if err != nil {
        fmt.Printf("error when dumping publickey: %s \n", err)
        os.Exit(1)
    }
    publicKeyBlock := &pem.Block{
        Type:  "PUBLIC KEY",
        Bytes: publicKeyBytes,
    }
    publicPem, err := os.Create("../public.pem")
    if err != nil {
        fmt.Printf("error when create public.pem: %s \n", err)
        os.Exit(1)
    }

Code for generating Jwt token with private.pem cerificate



func GenerateAccessToken(name, Id string) (string, error) {
    privateKey, err := ioutil.ReadFile("private.pem")
    if err != nil {
        log.Fatal(err)
    }
    rsaprivateKey, err := jwt.ParseRSAPrivateKeyFromPEM(privateKey)
    if err != nil {
        log.Fatal(err)
    }


    token := jwt.NewWithClaims(jwt.SigningMethodRS256, jwt.MapClaims{
        "name": name,
        "ID":   Id,
        "exp":  time.Now().Add(10 * time.Minute).Unix(),
    })


    tokenString, err := token.SignedString(rsaprivateKey)
    if err != nil {
        return "", errors.New("failed to create Token")
    }
    return tokenString, nil
}

We can now verify the jwt token with public.pem certificate. We can further make a middleware functions to verify jwt token.



func ValidateJwt(signedToken string) (*jwt.Token, jwt.MapClaims, error) {

    // Load the RSA public key from a file

    publicKey, err := ioutil.ReadFile("../public.pem")

    if err != nil {

        log.Fatal(err)

    }

    rsaPublicKey, err := jwt.ParseRSAPublicKeyFromPEM(publicKey)

    if err != nil {

        log.Fatal(err)

    }

<span class="c">// Parse the signed JWT and verify it with the RSA public key</span>
<span class="n">token</span><span class="p">,</span> <span class="n">err</span> <span class="o">:=</span> <span class="n">jwt</span><span class="o">.</span><span class="n">Parse</span><span class="p">(</span><span class="n">signedToken</span><span class="p">,</span> <span class="k">func</span><span class="p">(</span><span class="n">token</span> <span class="o">*</span><span class="n">jwt</span><span class="o">.</span><span class="n">Token</span><span class="p">)</span> <span class="p">(</span><span class="k">interface</span><span class="p">{},</span> <span class="kt">error</span><span class="p">)</span> <span class="p">{</span>
    <span class="k">if</span> <span class="n">_</span><span class="p">,</span> <span class="n">ok</span> <span class="o">:=</span> <span class="n">token</span><span class="o">.</span><span class="n">Method</span><span class="o">.</span><span class="p">(</span><span class="o">*</span><span class="n">jwt</span><span class="o">.</span><span class="n">SigningMethodRSA</span><span class="p">);</span> <span class="o">!</span><span class="n">ok</span> <span class="p">{</span>
        <span class="k">return</span> <span class="no">nil</span><span class="p">,</span> <span class="n">fmt</span><span class="o">.</span><span class="n">Errorf</span><span class="p">(</span><span class="s">"unexepcted signing method: %v"</span><span class="p">,</span> <span class="n">token</span><span class="o">.</span><span class="n">Header</span><span class="p">[</span><span class="s">"alg"</span><span class="p">])</span>
    <span class="p">}</span>
    <span class="k">return</span> <span class="n">rsaPublicKey</span><span class="p">,</span> <span class="no">nil</span>
<span class="p">})</span>

<span class="n">claims</span><span class="p">,</span> <span class="n">ok</span> <span class="o">:=</span> <span class="n">token</span><span class="o">.</span><span class="n">Claims</span><span class="o">.</span><span class="p">(</span><span class="n">jwt</span><span class="o">.</span><span class="n">MapClaims</span><span class="p">)</span>
<span class="k">if</span> <span class="n">ok</span> <span class="o">&amp;&amp;</span> <span class="n">token</span><span class="o">.</span><span class="n">Valid</span> <span class="p">{</span>
    <span class="k">return</span> <span class="n">token</span><span class="p">,</span> <span class="n">claims</span><span class="p">,</span> <span class="no">nil</span>
<span class="p">}</span>
<span class="k">return</span> <span class="no">nil</span><span class="p">,</span> <span class="no">nil</span><span class="p">,</span> <span class="n">err</span>



}

Tweet Service

The Tweet Service is responsible for posting tweets and pushing them to a message queue for consumption by the timeline service. Additionally, the service is responsible for handling tweet interactions such as retweeting and liking. Retweets are also pushed to the message queue to ensure that followers can see them.

By utilizing a Message Queue, tweets can be processed efficiently and in real-time. RabbitMq was used as the message queue in the development of this project.

Prior to sending tweets to the message queue, they are stored in Postgres and Redis. Redis is used to store a copy of the tweet, along with information about users who liked and retweeted the tweet. User tweets are stored in a sorted set with a key format of "users:${username}:tweets", facilitating efficient data organization and retrieval.

Timeline Service

The Timeline service is tasked with fetching both the user timeline and home timeline. To handle incoming tweets obtained from the MQ, we will utilize a separate goroutine. Our approach will involve iterating through each new message, and if the author's number of followers is below x, we will employ the Push method. Otherwise, we will opt for the Pull method.

Push Method

In this approach the tweet will be send to every followers of the author. This will be stored in followers timeline in Redis with a key similar to "users:${username}:timeline".

What is the problem with this approach?

Suppose a tweet author has 1,000,000 followers so with this approach timeline service would have to send this tweet to 1million users. Even for the Blazing fast Redis that will take time to finish. In this scenario we opt for Pull method.

Pull Method

When the number of followers exceeds x, the Pull method is more efficient, and Twitter uses it for handling tweets from celebrities. If a user requests to load their timeline, we will first check if they follow any celebrities this can be done by maintaining a list of celebrity a user follows. If they do, we will pull the latest tweets from the celebrities and merge them with the user's timeline, returning the resulting tweets. This approach ensures that only active users receive the data, like a request and serve mechanism. In contrast, the Push method pushes tweets to both active and inactive users, resulting in wasted bandwidth.

Load Balancer

Since we have multiple services we want to a way to direct the request to each service, here comes the load balancer. Haproxy is used as the load Balancer in the development of this project. Haproxy directs the request it gets to appropriate services.

Service Registry

Every Service on startup register itself to service register In this case I've used Consul as the service registry. I configured consul in a way that after every 10s it makes a request to particular endpoint if it failed to respond it means our service is down. Consul provides us a Web interface where we can see if the services are online or not.

Web Interface: localhost:8500

Docker

My Twitter-Backend has been dockerized so you run it with just one command. Docker helped me setup this project easily. For those who don't know docker enables you to run any database, load balancer or cache with just 1-2 commands. You can also dockerize your application to ship it to production.

Conclusion

This project was fun to do and I've learned a lot by building this project hope you have learned something new. Do tell me any mistakes I made or how it can be improved. Be sure to check out my project on GitHub (Documentation available) and also check out thread I made on Twitter🙂.

Github: https://github.com/leoantony72/twitter-backend

Twitter: https://twitter.com/serpico_z/status/1605899167439757312

Intro to common components of System Architecture!

Leo Antony — Tue, 29 Nov 2022 14:35:16 +0000

1.Intro

This is a shot blog on most common system design components you will hear. You will be learning Load-balancers, Caching and Monolithic & Microservice Architectecture

2.Components

I introduce you to 10 most common terms you will hear if you are learning System Architecture. Below are some of the most important concepts you need to know if you getting into System Architecture.👇

1.Load Balancer

A single server can't handle millions of requests at a time. So, What's the solution? Add more servers but how do we route the users to them?🤔. Here a load balancer distributes the load among the available servers. Nginx/HaProxy are some popular load balancers.

2.Caching

Caching Layer is between the API and the database. When a request to retrieve a piece of data is sent the API checks the cache and if it's available it returns it to the user or else queries the database. Using a caching layer increases the speed of the read operation.

3.Monolithic

It is one large codebase with all the functions of your API tightly linked together. If a part of your system breaks your whole API will be down📉. Most project starts with a monolithic approach and then moves on to microservice architecture depending on their team size.

4.Microservice

Independent and contained services with each service having an individual database. For ex: if the timeline service fails it does not affect the tweet service or any other service. There will be a load balancer that routes the request to desired services. Microservice architecture introduces complexity in deploying and increases network latency. Each service can talk to each other using a Message Queue like kafka.

5.Scaling

Large enterprises have huge amounts of data. with the increasing amount of data each day their server should grow accordingly.
There are two types of scaling horizontal & vertical scaling

Vertical Scaling - It means upgrading the current system.upgrading RAM, CPU’s and hard disk drives but there is always a certain limit on how much we can upgrade.
Horizontal Scaling - It means to add multiple systems to improve the performance of the service. A load balancer is needed to distribute the load to multiple systems.

Conclusion

This was a fairly a short blog, hope you have learned something new in this. next time I am going to be building Twitter back-end clone🔥.

Distributed Chat Application

Leo Antony — Fri, 04 Nov 2022 15:38:35 +0000

1.Introduction

Building a distributed chat system was one of the things i wanted to build, Designing a chat system like slack or messenger has always been among the top questions asked in interview's. I am self taught developer currently in high school building side projects. In this article, I want to share with you my design of Distributed Chat Application.

1.1.Prerequisite

Basic knowledge of Load balancer
How Websockets Works
Understanding Message Queue
Cassandra

2.Features

At First we must list out the features that we are going to build as this gives us a better understanding at the project.

One on One message: Two users can chat with each other
Group Chat: users can send message to a group
Create Groups: users can create groups
Join Groups: users can join group with their name
servers must be able to scale: Multiple web-server instance behind the load balancer.

3.Database Design

We would use NoSQL database such as cassandra. It's easy to scale up and it's very suitable for write-heavy works.

3.1 Database Schema

In Cassandra performing JOINS is not possible, That's a feature🐱‍💻

In here Timestamp is maintained in each message to retrieve message when user logs in.

4.System Design

One of our core feature will be scalability so We will be deploying 3 instance of our API/WebSocket server and with the help of a load balancer like Nginx/haproxy traffic will be distributed to each server.

4.1.Private Chat

What are the complexity that we deal when we spin up more than 1 instance?

Suppose user1 has to send a message to user2 but due to the existence of 3 servers User1 establish a websocket connection with S1 and User2 establish a connection with S3. How the heck am i supposed to send message to User2 now?😵

Introducing distributed servers == Increases Complexity

My approach to the solution is to keep a Message Queue, you can choose between kafka, RabbitMQ or Redis. I chose Redis because it's less Complexity and sub millisecond write and read speed🚀. Each server will have it's own topic in redis. So when a message needs to be send to other server it will be send to respective server's topic. Each servers will also consume new message from their topic.

How do i know which server the receiver is connected to?

We will maintain a User Mapping table with columns userID and SeverID. When users connect to the websocket server their userid and serverid is stored in the User Mapping Table in cassandra and when user disconnects their record is deleted or marked disconnected by introducing an additional field status.

This will be our final solution for private chat below is the data flow of this design👇
Data Flow:

User1 and User2 sends request to load-balancer
load-balancer connects User1 to S1 & User2 to S3
User1 wants to send message to User2
User1 sends the message to S1
S1 retrieve's which server User2 is connected to from User Mapping Table
S1 sends the message to S3 TOPIC in redis Queue
S3 retrieve's new message from S3 TOPIC from redis Queue
S3 then sends the message to User2

4.2.Group Chat

Although Group chat follows the same design we can make some tweaks in the websocket servers for better performance.

What is the performance issue that we are going to encounter in this architecture?

Assume that User1(Connected to S1) needs to send a message to group X with 400 members, These members are distributed among the other two servers S2 & S3 with 200 members connected to each server(assuming all the users are active right now).S1 would get the list of members in the group X then loop through each of them and send the message to the server member are connected to. This would send 200 Duplicate message to S2 & S3, Therefore this is not an optimal case for us.😕

This is how i prevented the sending of duplicate messages, At First we retrieve the group members from cassandra then get the SERVERID of the servers to which they are connected from user mapping table. We then group them according to their SERVERID with SERVERID being the key and array of USERID being the value. We then loop through the keys and send to the respective Topics in Message Queue according to their key with the message.

isgroup field is used to check if the message is being sent to a group or not, group_members is an array of members of group X which are connected to server S2 in this case.

Data Flow:

User1 connects to S1
User1 sends a message to group "X"
S1 fetches the group members of group "X"
S1 loops through them to get the id of their connected server
Groups the members based on the server they are connected on
serverId being the key and array of users being the value
loops through the keys and sends the message to the Topic in message queue with the same key along with its value (array of members)
S2 & S3 receives the message with server_id, sender, is_group, group_name, group_members and message
S2/S3 loops through the group members (users which belong to group X which are connected to S2/S3) and sends the message to them

This is the solution that i came up with and this might not be the perfect solution and lot of other improvements can be made in this. If you have a better suggestion please let me know below I would love to discuss it😁. This was made after i finished making a distributed chat system with golang, redis and cassandra using redis as Message Queue.

Distributed Chat System: https://github.com/leoantony72/go-chat2
Twitter: https://twitter.com/serpico_z

If this post gets good attraction i will do system design of notification system next time🐱‍💻. See you next time👋

E-Commerce API Made With Node-Js,Typescript,PostgreSQL,Redis And Kafka

Leo Antony — Wed, 09 Mar 2022 15:18:41 +0000

Introduction

E-commerce API made with typescript, express ,PostgreSQL,Redis & Kafka with all the basic features and more !!

Source Code - https://github.com/leoantony72/E-commerce-Api

Environment variables

PORT = 4000
DB_USER = PGUSER
DB_PASSWORD = PGPASSWORD
DB_HOST = postgresdb
DB_PORT = 5432
DB_DATABASE = e_commerce
SESSION_SECRET = Session Secret
SESSION_MAXAGE = 3600000 * 60 * 10
GMAIL_USER = email // eg:(test123)
GMAIL_PASS = Gmail password
STRIPE_PUBLIC_KEY = Stripe public Key
STRIPE_SECRET_KEY = Stripe Private Key

Docker build

docker-compose up // in the root dir

Setup Postgres,Redis & Kafka

⭕*Postgres *

docker exec -it postgresdb psql -U username -W e_commerce

Copy and paste db.sql file from db_model

⭕*Redis *

docker exec -it redis-e-commerce bash -c redis-cli

paste this command

BF.RESERVE usernames 0.00001 10000

⭕Kafka

docker exec -it broker bash

kafka-topics --create --bootstrap-server localhost:9092 --replication-factor 1  --partitions 1  --topic orders

With this our setup is finished , all the Api routes is


1. /api/store                       GET     Access the Store
2. /api/auth/register               POST    Register a User
3. /api/auth/login                  POST    Login A user
4. /api/auth/logout                 POST    Logout A User
5. /api/auth/forgotpassword         POST    Req For Password Reset
6. /api/auth/reset-password/        POST    Resets The Password
7. /api/verify                      GET     Verify The Email
8. /api/admin/product               POST    Insert A Product
9. /api/admin/product/:id           PUT     Update Product
10. /api/admin/product/:id          DELETE  Delete A Product
11. /api/admin/stock/:id            PUT     Update Stock
12. /api/admin/discount/:pid        GET     Get Discount of a Product
13. /api/admin/add_discount/:id     POST    Add Discount to a product
14. /api/admin/activate/:id         GET     Activate Discount
15. /api/manager/orders             GET     Gets Order
16. /api/manager/order/:orderid     GET     Get Individual Order Detailes
17. /api/shipper/updateorder/:oid   POST    Sents Confirm Order Request
18. /api/order/confirmdelivery      POST    User Confirms Delivery
19. /api/products                   GET     Get Products
20. /api/products/:pid              GET     Get Individual product
21. /api/checkusername              POST    Checks If Username is Taken
22. /api/ratings/:pid               GET     Get Rating
23. /api/ratings/:pid               POST    Add Rating To Product
24. /api/ratings/:pid               PUT     Update Rating
25. /api/ratings/:pid               DELETE  Delete Rating
26. /api/cart                       GET     Get Cart Items
27. /api/addItem/:pid               POST    Add Item To Cart
28. /api/removeItem/:pid            DELETE  DEL Item In Cart
29. /api/purchase                   POST    Purachse Item
30. /api/userAddress                POST    Add User Address
31. /api/userAddress                DELETE  DEL User Address
32. /api/userAddress                PUT     Update User Address

More details on the routes are provided in the documentation 👉

Source Code: https://github.com/leoantony72/E-commerce-Api

Thank you for your time ...😁

Lava lamps securing the Web?🤷‍♂️

Leo Antony — Tue, 08 Mar 2022 12:37:22 +0000

computer generates random number to be used for encryption key, The more random they are the more secure.computer are made to provide predictable output based on given inputs so it isn't suitable for generating encryption key.Nowadays computers have secure lib that provide CSPRNG(Cryptographically-secure pseudorandom number generator) So it's suitable for using as key.

computers can also be fed with real word random data because events in the physical world are unpredictable and one of the sources of randomness cloudflare uses is Lava Lamps.

Cloudflare have about 100 lava lamps with camera pointed to them and at regular interval they take pictures and send them to cloudflare servers. Images are stored as series of numbers with each pixel having its value ,So each images becomes a string of total random numbers .This becomes their base to encrypt the internet.

Cryptographic seed - the data that a CSPRNG program starts with for generating random data.

*CSPRNG **needs **seed *(cryptographic seed) as a starting point from which to produce more random data.Clouflare uses a CSPRNG to generate random key with data collected from lava lamps as the seed.

that's it for this blog , see you next time with more interesting stuff...