Forem: LinceMathew

Searching Billions in Seconds: How HNSW Solved the Scale Problem

LinceMathew — Mon, 27 Apr 2026 08:10:56 +0000

We have a massive problem with how computers find things. If you have a few hundred photos on your phone, you can find one instantly, but trying to find one specific item out of a billion creates a massive technical strain.

Most systems rely on "Linear Search", which is like looking through every single page of a ten-million-page book to find one word.

This "one-by-one" approach makes real-time tools like chatbots or movie recommendations grind to a halt as the data grows. Furthermore, modern data like images or text is "high-dimensional," which breaks traditional filing systems and makes them no faster than checking every item manually.

To fix this, researchers Yu. A. Malkov and D. A. Yashunin changed the rules of the game. They realized that we don't always need a "perfect" match if it takes an hour to find; we often just need a "good enough" match found in a millisecond.

In this article, we are diving into their research paper: "Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs."

We will explore how they used a multi-layered "highway system" to make searching a billion items feel as fast as searching a hundred.

Two Methods for Fast Searching

To solve the slowness problem, the researchers combined two concepts from computer science. The first is the "Small World" phenomenon.

You’ve likely heard of "six degrees of separation", the idea that you are connected to anyone on Earth through a short chain of acquaintances.

HNSW treats data the same way. It builds a map where every data point is a "node," and similar points are connected like friends. By jumping from "friend to friend" toward your target, you can navigate a massive dataset in just a few steps.

The second concept is Hierachy, which is inspired by a structure called a Skip List.

Imagine a high-rise building where the elevators only stop at every 10th floor. To get to floor 42, you take the express elevator to floor 40 and then walk down to 42. HNSW creates "express layers" at the top with only a few data points and long-range connections.

These top layers allow the search to "skip" across the entire database to the right neighborhood instantly, before dropping down to the ground floor to find the exact result.

The Multi-Layer Highway System

To bridge the gap between speed and accuracy, the researchers introduced a multi-layer structure called a Hierarchical Navigable Small World (HNSW) graph.

You can think of this as a "highway system" for data. Instead of one giant, flat network where everything is connected, HNSW organizes data into layers.

Top Layers (Express Lanes)

These layers contain only a few data points with long-distance connections. Much like an express highway between major cities, they allow the search to "skip" across the entire dataset to reach the right neighborhood in just a few jumps.

Bottom Layers (Local Streets)

As the search moves down through the hierarchy, the layers become denser with shorter connections. Once the search reaches the ground layer—which contains every single piece of data—it uses these "local streets" to fine-tune the results and find the exact match.

This hierarchy is inspired by a data structure called a Skip List, but instead of simple linked lists, it uses complex proximity graphs.

By separating links by their "distance scales," the algorithm ensures that the search time scales logarithmically. This means that even if the dataset grows from thousands to billions, the number of steps required to find an answer stays manageable and fast.

Balancing Speed and Accuracy

To make this system work, there are a few settings that determine how the graph is built and searched.

Connections per point (M): This is the number of "links" each piece of data creates with its neighbors. If you give each point more connections, the map becomes more detailed and accurate, but it also takes up more memory.
The Build Effort (efConstruction): This determines how much work the system does when first creating the index. A higher setting means the "highways" and "local streets" are better connected, making future searches much more reliable.
The Search Depth (ef): This is a setting you use during the actual search. It tells the computer how many neighbors to check before it decides it has found the best match. You can turn this up at any time to get better results without having to rebuild your entire database.

Building an HNSW Index

To see how this works, we can use a popular Python library called hnswlib. This is the same code used in the original research to prove how fast the system is. The following example shows how to set up a database and perform a search in just a few lines.

Import the Tools

We start by importing hnswlib to build the graph and numpy to handle the coordinates of our "meaning map."

import hnswlib
import numpy as np

Create the "Meaning Map"

Let's give our items coordinates based on two features: Sweetness and Size. Fruits will have high sweetness and low size, while furniture will have low sweetness and high size.

data_labels = ["Apple", "Banana", "Cherry", "Chair", "Table", "Sofa"]
data_vectors = np.array([
    [0.9, 0.2], # Apple 
    [0.8, 0.3], # Banana
    [0.9, 0.1], # Cherry
    [0.1, 0.7], # Chair
    [0.1, 0.9], # Table
    [0.2, 0.9]  # Sofa 
], dtype=np.float32)

dim = 2 # Only 2 features: Sweetness and Size
num_elements = len(data_vectors)

The data now looks like a list of coordinates. For example, Apple is at [0.9, 0.2].

Choose the Measuring Tape

We initialize the index. We use 'l2' (Euclidean distance) to measure similarity. On our map, this is just a straight line between two points; the shorter the line, the more similar the items are.

p = hnswlib.Index(space='l2', dim=dim)

Build the Highway System

We set the rules for the HNSW structure. M sets the number of connections for each point, and ef_construction tells the system how hard to work to find the best neighbors when first building the "highways."

p.init_index(max_elements=num_elements, ef_construction=100, M=16)

Index the Data

This step takes our fruits and furniture and organizes them into layers. The "High-Sweetness" items (fruits) will naturally end up in one neighborhood, while the furniture ends up in another.

p.add_items(data_vectors)

Perform the Search

Now we search for an "Orange". We haven't told the system what an orange is, but we give it the coordinates 0.85, 0.25. We tell the system to find the 3 closest neighbors.

query_vector = np.array([[0.85, 0.25]], dtype=np.float32)
p.set_ef(10)
labels, distances = p.knn_query(query_vector, k=3)

The system returns the IDs for Apple, Banana, and Cherry. Even though "Orange" wasn't in our database, HNSW used the highway system to skip past the "Furniture" neighborhood and find the items that lived in the same "Sweet and Small" neighborhood.

here is the layered flow diagram.

In this diagram you can see there are three layers in the graph. The top layer has only a few nodes and the bottom layer has all the nodes. The search starts from the top layer and moves down to the bottom layer.

Realworld Applications of HNSW

The reason this research paper is so influential is that it solved the "scale" problem for some of the most popular technology today. Because HNSW can find a "neighborhood" of similar ideas in milliseconds, it is used in:

AI Chatbots & RAG

When you ask an AI a question, it uses HNSW to search through millions of documents to find the specific paragraphs that contain the answer.

Recommendation Engines

Apps like Spotify or Netflix use this logic to find songs or movies that are "mathematically close" to what you just finished watching.

Image Search

Tools like Google Lens compare the "fingerprint" of your photo against billions of others to find a match instantly.

Fraud Detection

Banks use it to see if a new transaction looks "similar" to known patterns of theft or if it fits your normal "neighborhood" of spending.

Conclusion

Before this paper, we had a difficult choice: we could have a search that was perfectly accurate but incredibly slow, or a search that was fast but broke down as it grew.

Malkov and Yashunin’s HNSW changed that by introducing the "highway" system for data. By accepting a tiny bit of approximation, they gave us a way to search through billions of items in the blink of an eye.

Automate the Boring Stuff! Check Out This Open-Source Tool

LinceMathew — Sun, 13 Jul 2025 16:09:50 +0000

If you’re tired of doing the same tasks over and over—copying data between apps, checking emails, or updating spreadsheets—there’s a better way. Meet n8n, a tool that helps you automate those repetitive jobs without needing to be a programmer.

n8n stands for Node Automation. It’s an open-source platform that lets you connect different apps, move data between them, and build powerful automations.

You can think of it like Zapier, but with more flexibility. Plus, you can host it yourself, which means more control and no monthly fees.

Set It Up, No Sweat

Getting started with n8n is simple, and there are a few different ways to run it depending on your preference. Whether you're just trying it out or setting it up for daily use, here are your options:

Option 1: Run n8n with Docker
If you have Docker installed, this is one of the easiest and cleanest ways to run n8n. Open your terminal and run:

sudo docker run -it \
  -p 5678:5678 \
  -v ~/.n8n:/home/node/.n8n \
  -e N8N_BASIC_AUTH_USER=user \
  -e N8N_BASIC_AUTH_PASSWORD=pass \
  n8nio/n8n

This will start n8n on port 5678 with basic username and password protection. Your workflows will be saved in your local .n8n folder.

Option 2: Use npx to Try n8n Instantly
If you have Node.js, you can launch it using just one command:

npx n8n

That’s it—no setup, no extra install. It’s great for quick testing.

Build Your First Workflow

In this example, we’ll create an automation that fetches the daily trending Go (Golang) repositories on GitHub and adds them to a Google Sheets spreadsheet. This shows how n8n can help you collect and organize information automatically.

In n8n, a workflow is a series of connected tasks that run automatically, and each task is called a node. Nodes perform specific actions like getting data, processing it, or saving results.

Here’s a quick look at the workflow we’ll build:

Cron Node: This triggers the workflow to run once every day (or whenever you want).

HTTP Request Node: It fetches the trending repositories page from GitHub for the Go language.

HTML Extract Node: This node reads the web page and pulls out the repository names and descriptions.

Function Node: It organizes the data into a clean format with title, description, and URL.

Google Sheets Node: Finally, this node adds the data into a Google Sheets spreadsheet.

Next, we’ll walk through each step so you can build this workflow yourself.

Prerequisites

Before building the workflow, make sure you have a Google Sheet ready. The first row (header) of the sheet should have these three columns:

Title | Description | URL

You also need to authorize Google Sheets in n8n. To do this:

Go to Credentials → Google Sheets OAuth2 in n8n.
Create a Google Cloud project and set up an OAuth Client ID.
Add these credentials to the Google Sheets node in your workflow.

1. Cron Node — Trigger Your Workflow

The Cron node triggers your workflow based on time. You can set it to run daily, hourly, or whenever you want. For this example, set it to run once a day.

2. HTTP Request Node — Fetch GitHub Trending Repos

This node sends a GET request to GitHub’s trending page for Go repositories. The URL is:

https://github.com/trending/go?since=daily

You can replace go with any language you want to track. The response will be the page’s HTML in text format.

3. HTML Extract Node — Get Repo Names and Descriptions

The HTML Extract node reads the page’s HTML and extracts the repository names and descriptions using CSS selectors.

For repo names:

article.Box-row h2 a

For descriptions:

article.Box-row p

The node uses the HTML body from the HTTP Request node as input.

4. Function Node — Format Data for Google Sheets

This node takes the extracted data and cleans it up. It trims extra spaces and creates a neat JSON object with three fields: Title, Description, and URL. The URL is built by adding https://github.com in front of the repo name.

5. Google Sheets Node — Save Data to Your Sheet

Finally, this node adds the formatted data to your Google Sheet. Use the Append operation and specify your sheet’s ID and the range (for example, Sheet1!A:C). Map the fields Title, Description, and URL to the corresponding columns in your sheet.

This is how our final workflow looks.

Now, let’s trigger the workflow manually.
Boom! All the trending Go repositories from GitHub are added to our Google Sheet!

What Else Can You Automate with n8n?

n8n is very flexible and can help automate many everyday tasks. Here are some common ways people use it:

Notifications
For example, when a new order comes in, n8n can automatically send a message to your team on Slack and send an email alert. This keeps everyone updated without manual effort.

Reporting
You can gather important data or metrics every week, create a report, and have n8n email it to your team before meetings. This saves time and ensures everyone has the latest information.

Web Scraping
Want to track competitor prices? n8n can scrape product prices from websites and log the data into a Google Sheet for easy comparison.

These examples show just a few of the many ways n8n can simplify your work by automating repetitive tasks.

The Bottom Line: Is n8n for You?

If you find yourself doing the same tasks over and over—copying data, sending emails, checking updates—n8n can save you a lot of time. It’s powerful, flexible, and doesn’t require you to be a developer.

Whether you're automating small personal tasks or building complex workflows for your team, n8n makes it easier. And the best part? You’re in full control, especially if you choose to self-host it.

PRO TIP: Struggling to locate API endpoints across your codebase?

LiveAPI lets you instantly find and explore every endpoint across all your repositories — give it a try and see how much easier navigation can be!

Getting the Best Out of GitHub Copilot

LinceMathew — Sun, 01 Jun 2025 16:20:14 +0000

As my first in-editor AI coding partner, I started with Cursor. However, four months ago, I decided to switch to GitHub Copilot. While using Cursor, I was aware of Copilot and its features, but at the time, it felt unstable and not as polished as Cursor. Recently, I noticed that GitHub Copilot has been improving significantly, becoming a strong competitor to Cursor. This encouraged me to give Copilot a try, especially since it offers a larger quota for expensive LLM calls and has proven to be stable even in its preview version.

Another reason for the switch was my existing GitHub Pro subscription, which meant I could save $22 per month by not needing Cursor’s subscription if Copilot proved to be a good alternative.

In this article, we’ll explore some of unique Copilot’s features i found and best practices we can use to make the best out of copilot.
FYI:My primary focus is on Github copilot extension on VScode insider, even you can access copilot from web version.

Personalizing GitHub Copilot Responses

GitHub Copilot allows you to customize its responses to better suit your needs.Its difficult and irritating to mention stick to functional programming, use early return etc in every chat completion. These custom instructions help Copilot understand your preferences, project requirements, or even organization-wide standards. There are three types of custom instructions you can use:

Personal Custom Instructions: Copilot’s responses to your individual preferences across all conversations.
Repository Custom Instructions: Define project-specific guidelines, frameworks, or coding standards for a particular repository.
Organization Custom Instructions: Apply organization-wide preferences, managed by organization owners, to ensure consistency across all repositories.

Let’s start with personal custom instructions.

Adding Personal Custom Instructions

Personal custom instructions let you customize how GitHub Copilot responds to you, making it more aligned with your preferences. Setting them up is straightforward and can be done either in the Copilot Chat panel or immersive mode.

How to Add Personal Instructions

Open the Copilot Chat panel or immersive mode:

Click the Copilot Chat icon in the top-right corner of any GitHub page to open the panel.
If you prefer a full-page view, click the dropdown next to the Copilot Chat icon and select "Immersive." You can also go directly to https://github.com/copilot.

Access Personal Instructions:

In the Chat panel or immersive page, click the dropdown menu at the top-right corner and select "Personal instructions."

Add Your Preferences:

Write your preferences in the text box using natural language. You can format them however you like—either as a single block of text, line by line, or separated by blank lines.
If you’re unsure where to start, click the templates button to see examples of common instructions. For instance, selecting "Communication" will add placeholders like {format} that you can customize.

Save Your Instructions:
- Click "Save" to activate your instructions. They’ll stay active until you decide to change or remove them.

Example

Let’s say you want Copilot to respond in Portuguese and keep explanations short and simple. Here’s how you can set it up:

Open the Copilot Chat panel.
In the "Personal instructions" section, add the following: Always respond in Portuguese. Keep explanations concise and break them down step by step.
Save your instructions.

Now, whenever you interact with Copilot, it will respond in Portuguese and provide clear, concise explanations tailored to your preferences.

continue reading the full article here.

Data Races in Go: What They Are and Why You Should Care

LinceMathew — Sun, 13 Apr 2025 16:04:16 +0000

A data race happens when two or more goroutines access the same memory location at the same time, and at least one of them writes to it — without proper synchronization.

This can lead to unpredictable behavior. Your program might sometimes work just fine, and other times crash or return wrong results. That’s because the order in which goroutines run isn’t guaranteed, so without coordination, one goroutine might read a value while another is in the middle of changing it.

Why Are Data Races a Problem?

Data races aren’t just theoretical—they can cause real, unpredictable problems that are difficult to catch and fix. Here’s a simple example:

func race() {
    wait := make(chan struct{})
    n := 0

    go func() {
        n++ // read, increment, write
        close(wait)
    }()

    n++ // conflicting access
    <-wait
    fmt.Println(n)
}

In this code, both the main goroutine and the anonymous goroutine are updating the same variable n. Since there’s no coordination between them—no locks, no channels protecting the access—we don’t know which n++ happens first. The output might be 1, 2, or something even weirder depending on how the Go runtime schedules the goroutines.

This is the essence of a data race: multiple goroutines accessing shared memory without synchronization.

So what’s the big deal?
The biggest problem with data races is that they don’t fail consistently. Your code might run fine 100 times and crash on the 101st. That makes them:

Hard to reproduce
Hard to debug
Easy to miss in tests

But the impact can be serious:

You might get incorrect results or corrupted state
Goroutines might get stuck in deadlocks
Your program might even crash if memory is accessed in unsafe ways

If your app "sometimes fails for no reason," a data race could be the hidden culprit.

How to Spot Data Races

The good news? Go makes it easy to spot data races using a built-in tool: the race detector.
Let's try it out with our example:

func main() {
    var wg sync.WaitGroup
    counter := 0

    for i := 0; i < 10; i++ {
        wg.Add(1)
        go func() {
            counter++ // race condition
            wg.Done()
        }()
    }

    wg.Wait()
    fmt.Println(counter)
}

Each goroutine tries to increment counter. But since there's no synchronization—no mutex, no channel—they can interfere with each other. The result is unpredictable, and chances are the final output won’t be 10.

To catch this, run the program with Go’s race detector:

go run -race main.go

The -race flag enables Go’s runtime race detector. If there’s a data race, you’ll see a warning like this:

This output shows exactly where the race occurred, what variable was involved, and which goroutines were part of the conflict. It’s a powerful way to catch concurrency issues early—ideally before they make it into production.

How to Solve Data Races

Once you’ve spotted a data race, the next step is fixing it. Go gives you three main tools for this: mutexes, atomic operations, and channels. Each has its place, depending on what you're doing and how you want to coordinate access to shared data.

Using a Mutex

A sync.Mutex ensures that only one goroutine can access a piece of code at a time. You lock before accessing the shared variable and unlock after you're done.

var counter int
var mu sync.Mutex
var wg sync.WaitGroup

wg.Add(2)

go func() {
    mu.Lock()
    counter++
    mu.Unlock()
    wg.Done()
}()

go func() {
    mu.Lock()
    counter++
    mu.Unlock()
    wg.Done()
}()

wg.Wait()
fmt.Println("Counter:", counter)

Here, the mutex (mu) protects access to counter. Only one goroutine can increment it at a time. It’s simple and effective—great for most shared state scenarios.

Using sync/atomic

For basic numeric operations (like counters or flags), the sync/atomic package provides a faster, lock-free alternative.

import "sync/atomic"

var counter int64
var wg sync.WaitGroup

wg.Add(2)

go func() {
    atomic.AddInt64(&counter, 1)
    wg.Done()
}()

go func() {
    atomic.AddInt64(&counter, 1)
    wg.Done()
}()

wg.Wait()
fmt.Println("Counter:", counter)

Atomic operations are lightweight and efficient—but they only work for specific use cases, like updating integers or booleans. You can't use them for complex updates or data structures.

Using Channels

Channels are a Go-native way to synchronize access to data. Instead of locking, you "own" the value by passing it through a channel.

counter := make(chan int, 1)
counter <- 0 // initialize

go func() {
    val := <-counter
    val++
    counter <- val
}()

go func() {
    val := <-counter
    val++
    counter <- val
}()

time.Sleep(time.Second) // wait for goroutines
fmt.Println("Counter:", <-counter)

Here, the value lives inside the channel, and only one goroutine can access it at a time. It’s a safe and idiomatic way to coordinate shared state.

Which One Should You Use?

Use Mutex: when you need simple locking for shared resources.
Use Atomic: for fast, low-level operations on numbers.
Use Channels: when goroutines need to communicate, not just protect data.

Data races can quietly break your code in unpredictable ways. The good news? Go gives you simple tools to catch and fix them—like the -race flag, mutexes, channels, and atomic operations.

Just remember: when goroutines share data, coordinate access. Pick the right tool for the job, and you’ll avoid a whole class of sneaky bugs.

"Don’t communicate by sharing memory; share memory by communicating."
— Rob Pike

PRO TIP: Are you looking for an efficient way to create API documents for your projects?

LiveAPI get all your backend API documented in one shot Try it out and share your feedback!

Don't Let Your LLM Hallucinate—Check Out These Prompting Rules and Methods!

LinceMathew — Sun, 23 Feb 2025 16:12:46 +0000

From the LLM Bootcamp lectures by UC Berkeley PhD alumni, I’ve learned key rules and methods that improve prompts. This article covers techniques like Active Prompting, Meta Prompting, and Few-Shot Prompting etc designed to make models think more systematically and produce better results.

Essential Rules for Writing Effective Prompts

Here are a few important rules that can make prompts more effective. These techniques improve the accuracy and usefulness of responses.

Use Structured Text

Language models perform better when given well-organized input. Just like humans find clear instructions easier to follow, models also work better with structured prompts.

Example: Asking for a Python Function
❌ Without Structure:
"Write a Python function that checks if a number is prime."

✅ With Structure (Pseudocode):

Define a function is_prime(n):
  If n is less than 2, return False
  For each number i from 2 to sqrt(n):
    If n is divisible by i, return False
  Return True

A structured prompt makes it easier for the model to understand and follow specific steps, reducing mistakes.

Break Down Complex Requests (Decomposition)

When asking a model to complete a big task, it’s better to split it into smaller steps. This improves accuracy and keeps responses more focused.

Example: Writing a Blog Post
❌ All-in-One Prompt:
"Write a blog post about climate change and include recent statistics."

✅ Step-by-Step Approach:

Gather facts: "List recent climate change statistics from 2023."
Plan the content: "Create an outline for a climate change blog post."
Write in parts: "Write an engaging introduction for a climate change blog post."
Expand each section separately.
Breaking down the task ensures each part is handled properly before moving to the next.

✅ With Structure (Pseudocode):

Define a function is_prime(n):
  If n is less than 2, return False
  For each number i from 2 to sqrt(n):
    If n is divisible by i, return False
  Return True

Guide the Model’s Reasoning (Chain of Thought)

Instead of asking for a direct answer, guiding the model through logical steps leads to better responses.

Example: Checking for a Prime Number
❌ Simple Prompt:
"Is 2,345 a prime number?"

✅ Step-by-Step Prompt:
"Let’s determine if 2,345 is a prime number by checking divisibility step by step."

This approach encourages the model to explain its reasoning, making the response more reliable.

Combine Multiple Responses (Ensembling)

No model is perfect. Each has strengths and weaknesses, so combining multiple responses can lead to better accuracy.

For example, instead of relying on a single model to summarize an article, you can:

Ask two different models and compare results.
Run the same model multiple times and merge the best answers. This method helps filter out inconsistencies and improves response quality.

Different types of Prompting Methods

Active Prompting

Active Prompting is a method that helps language models improve their reasoning by selecting the most useful examples for learning. Instead of relying on a fixed set of pre-written prompts, this approach identifies uncertain or ambiguous questions and refines them through human annotation.

How Active Prompting Works

The process involves four key steps:

Uncertainty Estimation – The model answers a set of training questions multiple times (e.g., five attempts). If the answers vary significantly, the question is marked as uncertain.
Selection – The most uncertain questions are chosen for human review.
Annotation – Humans provide step-by-step reasoning for these selected questions.
Inference – The newly annotated examples are added back into the system, improving the model’s accuracy for similar tasks. #### Why Active Prompting Matters

Continue reading the complete article here: https://journal.hexmos.com/dont-let-your-llm-hallucinate-check-out-these-prompting-rules-and-methods/

Common API Design Mistakes and How to Avoid Them

LinceMathew — Tue, 31 Dec 2024 18:15:05 +0000

Maintaining and designing scalable APIs is a tremendous task, especially once you have more than 100 APIs for your project. In the era of API as a Service, knowledge of API design is unavoidable.

In this article, we will discuss a few RESTful API design standards you can apply to your projects for building an ideal API ecosystem.

Focus on Resources

The fundamental purpose of APIs is to provide access to particular resources. An API resource represents a piece of data or functionality that the API provides access to.

For example, in a CRM (Customer Relationship Management) application, typical resources include:

Contacts
Leads
Accounts
Tasks

A resource can be in the form of a singleton (a single account) or a collection (a group of tasks).

We can design an API in the format GET /contacts to fetch a list of all contacts or PUT /tasks/301 to update a task's details.

Proper URL Naming

Naming conventions are important for URLs. It is better to use nouns for RESTful URLs, as RESTful APIs are designed for collecting resources and communication, not for performing actions or operations.

Recommended Approach

GET /contacts

GET /tasks/:id

Less Preferred Approach

GET /get-contacts

GET /get-task-by-id/:id

Use Path Parameters

Using path parameters provides a clear and structured way to identify resources and their relationships within the API.

GET /tasks/:id

Avoid this

GET /tasks

{
 id:1
}

Maintain API hierarchy

Maintaining a well-defined resource hierarchy helps define relationships between different resources in an API.

In the case of CRM
GET /user/{userId}/task/{taskId}

Here, the API clearly fetches the task based on the task ID under a specific user.

API versioning

APIs will get modified as the product grows. So, modifications to the API should not affect older users, older clients, or frontend code. That's when versioning comes in.

Use API versioning when:

Renaming or changing any endpoint or property
Parameter changes
Modifying a data format

Versioning can be done by using URL Path Change or by using a Header.

URL Path
Version is defined in the path itself.
Example: https://example.com/api/v2/user

Header Versioning
Version defined in the Header of the API
Accept: version=1.0

Maintaining and designing APIs is a never-ending challenge. It will differ for different products, use cases, and contexts. This article covers a few aspects that will help you to build better API ecosystems.

My team and I are developing LiveAPI, a super convenient API doc generation tool. Using LiveAPI, you can create API documentation for any web project within minutes, with minimal effort.
Also, once you connect, LiveAPI will keep your API documentation up to date based on code changes.

Try LiveAPI and share your valuable feedback. Keep connected for more tech tips and articles.

Learn Golang in 2025, You Won't Regret It

LinceMathew — Sun, 29 Dec 2024 18:34:57 +0000

As I explained in previous articles, we are working on building LiveAPI, a super-convenient auto-API doc generation tool. LiveAPI's backend is in Golang, and I'm discovering the unique and cool features of Golang.

For those who don't know, Golang (Go) is a programming language designed at Google in 2009. It is syntactically similar to C.

Before working on Golang projects, I primarily used Node.js and Python web frameworks. For a beginner switching to Golang from another domain, it can be a little difficult initially, but once you practice and gain expertise, you won't leave.

My first Golang project was to convert a Python CLI tool, Glee, to Golang.

We found Python to be slower and maintaining a single compiled CLI binary to be very difficult, the binary sometimes not compatible with Mac OS. These and other issues forced us to switch to Golang. Here is a Reddit post on the story and the issues behind the switch.

In this article, I will explain the unique features in Golang that attracted me to it.

1. Goroutines - Lightweight Concurrency

func main() {
    go sayHello("World") // runs concurrently
    time.Sleep(1 * time.Second) 
}

func sayHello(name string) {
    fmt.Printf("Hello, %s!\n", name)
}

Think of goroutines like tiny workers that can do tasks independently. They're much lighter than traditional threads - you can create thousands of them without a performance hit.

2. Channels - Built-in Communication

func main() {
    messages := make(chan string)
    go func() { messages <- "ping" }()
    msg := <-messages
    fmt.Println(msg)
}

Channels are like pipes that let goroutines communicate safely. Imagine two people passing notes through a tube - one writes and puts it in, the other takes it out.

3. Defer - Cleanup Made Simple

func readFile() {
    file, err := os.Open("test.txt")
    defer file.Close() // Will run when function exits 
}

Defer is like setting a reminder for cleanup tasks. It's similar to writing a post-it note saying "don't forget to close the file" right when you open it.

4. Interface Implementation - Implicit Contracts

type Writer interface {
    Write([]byte) (int, error)
}

type ConsoleWriter struct{}

func (cw ConsoleWriter) Write(data []byte) (int, error) {
    return fmt.Println(string(data))
}

Go's interfaces are satisfied implicitly - if a type has the right methods, it automatically implements the interface. It's like joining a club: you don't need to formally declare membership; if you can do what the club requires, you're automatically in.

5. Multiple Return Values - Native Error Handling

func divide(a, b float64) (float64, error) {
    if b == 0 {
        return 0, errors.New("cannot divide by zero")
    }
    return a / b, nil
}

func main() {
    result, err := divide(10, 0)
    if err != nil {
        fmt.Println("Error:", err)
        return
    }
    fmt.Println("Result:", result)
}

Multiple return values make error handling natural and explicit in Go. It's like getting both a package and a receipt when shopping - you can check if everything is okay before proceeding. This pattern encourages developers to handle errors properly.

These are just a few features I like in Go. There are also others, like single executable binaries, faster performance, type inference, built-in testing support, and cross-compilation.

If you're making resolutions for 2025, add Golang to your list. You won't regret it. Thanks for reading! If you want to learn Golang by contributing to an open-source project, check out glee and Lama2.

What is Mutex and How to Use it in Golang?

LinceMathew — Sat, 28 Dec 2024 20:07:45 +0000

During the development of LiveAPI, an Auto API documentation generation tool, I needed to implement a robust queue mechanism that scaled based on the number of server machine cores. This was crucial to prevent excessive resource usage (memory and CPU) that could lead to resource starvation, crashes, and a poor user experience.

In this article, I'll explain how I utilized mutexes in Golang to address this challenge.

What is a Mutex?
In concurrent programming, a Mutex (Mutual Exclusion) is a locking mechanism that prevents race conditions by ensuring only one goroutine can access a shared resource at a time. It's akin to a key to a room – only one person can hold the key and enter at once.

Mutex Usage in Golang
Let's illustrate how a Mutex can manage concurrent job execution:
Go's sync package provides several primitives for synchronization, with Mutex being one of the most commonly used tools.

var (
    maxConcurrentJobs int
    activeJobs        int
    jobMutex          sync.Mutex
)

In this code, the activeJobs variable tracks the number of currently running jobs. Since multiple goroutines might attempt to modify this variable concurrently, leading to race conditions, we use a Mutex to synchronize access.

// Check if we can process more jobs
jobMutex.Lock()
if activeJobs >= maxConcurrentJobs {
    jobMutex.Unlock()
    // Wait before checking again
    time.Sleep(time.Second)
    continue
}
jobMutex.Unlock()

How a Mutex Works
Locking: The Lock() method acquires exclusive access to the critical section.

Unlocking: The Unlock() method releases the lock.

Critical Section: The code between Lock and Unlock where the **shared resource is accessed.

Types of Mutexes in Golang
sync.Mutex: This is the basic mutual exclusion lock in Go. It allows only one goroutine to access the critical section at a time.

type SafeCounter struct {
    mu    sync.Mutex
    count int
}

sync.RWMutex: This is a reader/writer mutex that allows multiple readers to access the shared resource simultaneously, but only one writer at a time.

var rwMutex sync.RWMutex
// Reader methods
rwMutex.RLock()   // Lock for reading
rwMutex.RUnlock() // Unlock for reading

// Writer methods
rwMutex.Lock()    // Lock for writing
rwMutex.Unlock()  // Unlock for writing

Mutexes are essential tools for managing shared resources in concurrent Go programs. They prevent race conditions and ensure data integrity by controlling access to critical sections of code.

Working on Redis streams? Don't forget these commands.

LinceMathew — Sat, 28 Dec 2024 01:31:13 +0000

I’ve been learning and implementing Redis streams for the past few days to set up a real-time communication system and queue management for our product LiveAPI, an auto API doc generation tool.

In this article, let us discuss a few Redis Stream commands that you need to be aware of while building efficient solutions using Redis Streams.

XADD: This command helps to add new entries to a stream. Example: XADD mystream * sensor-id 1234 temperature 19.8 humidity 43.5

This command adds a new entry to the stream 'mystream'. The * tells Redis to auto-generate the entry ID. Each entry contains multiple field-value pairs (sensor-id, temperature, humidity).

2.XREAD: This command reads entries from one or more streams.
Example: XREAD COUNT 2 STREAMS mystream 0

This reads 2 entries from 'mystream' starting from the beginning (ID 0).

3.XRANGE: This command returns entries within a specific ID range.
Example: XRANGE mystream 1641293000000-0 1641293060000-0

This command is used to get historical data within a specific range.

4.XGROUP CREATE: Creates a consumer group for stream processing.
Example: XGROUP CREATE mystream mygroup $

Creates a consumer group named 'mygroup' for 'mystream'. The $ means the group will only read new messages (from the last ID).

5.XREADGROUP: Reads from a stream as part of a consumer group.
Example: XREADGROUP GROUP mygroup consumer1 COUNT 1 STREAMS mystream >

This reads one unread message from 'mystream' as consumer1 in mygroup. The > means "give me new messages that haven’t been delivered to other consumers".

6.XCLAIM: Used to transfer ownership of pending messages.
Example: XCLAIM mystream mygroup consumer2 30000 1692312456878-0

This command will transfer ownership from one consumer to another within the same consumer group. This is particularly useful in handling failed consumers or rebalancing workloads. XCLAIM ensures no messages are lost if a consumer fails, enables work redistribution for better load balancing, and provides message recovery mechanisms.

I hope these few commands help to give you an idea about Redis streams and their commands. Share your valuable feedback.

I Created API Docs for 5 Open-source Projects Within 10 Minutes

LinceMathew — Wed, 25 Dec 2024 11:40:04 +0000

Creating and maintaining API documentation for any project is a tremendous task that eats up most of your time. However, we can't ignore this step in web development engineering because API docs play a significant role in collaboration, decision-making, and guidance.

For the last few months, my team and I have been working on developing LiveAPI, a super convenient tool for creating API Docs with the help of AI. Existing documentation tools, for example, Swagger, take time and effort from developers to set up and maintain. We saw an opportunity here and developed an easy doc generation tool that works within a few clicks.

Here, I will explain how I created API Docs using LiveAPI.

First, sign up for LiveAPI here.

Once logged in, find and select the "Create API Docs" option.

Paste the GitHub/GitLab URL of the open-source or your personal project.

Click the "Create Docs" button and allow a few minutes for the process to complete.

Once finished, the documentation will be hosted on LiveAPI, and you can access the link immediately.

Using this method, I created API docs for 5 open-source web projects within 10 minutes:

Mongo-express: A web-based MongoDB admin interface written with Node.js, Express, and Bootstrap 5

InvenTree: An open-source inventory management system that provides powerful low-level stock control and part tracking.

Freescout: FreeScout is the super lightweight and powerful free open source help desk and shared inbox built with PHP (Laravel framework).

BookStack: A platform for storing and organising information and documentation.

realworld-springboot-java: ReadWorld.io backend project using spring boot java using spring-security, spring-data-jpa

Try LiveAPI today to create API docs for your personal or open-source projects in minutes.

Want to learn more about LiveAPI's features? Check out our landing page here and find other blogs in my bio.

Follow me for more exciting product news and tips!

Redis Pub/Sub vs. Redis Streams: Choosing the Right Solution

LinceMathew — Mon, 23 Dec 2024 19:40:33 +0000

While working on building a Go binary tool ("Runner") for our product LiveAPI (the super convenient API doc generation tool), we needed to establish communication between LiveAPI's backend and Runner. We initially tried the Redis pub/sub mechanism but later realized it was insufficient for our use case.

The Problems with Redis Pub/Sub

Redis Pub/Sub follows a traditional publish-subscribe model: Publishers send messages to channels, and subscribers receive messages from the channels they are subscribed to.

This presented several problems for LiveAPI:

Duplicate Operations and Wasted Resources: If a user ran the same Runner on multiple devices, the subscription behavior of pub/sub would cause the same operations to run on multiple devices, creating duplication and wasting resources.

Memory and Data Loss: Pub/sub doesn't retain or persist messages, so if the server crashed or a job failed due to a panic, there was no way to restart the operations.

Lack of Acknowledgement: The LiveAPI backend had no indication that a job request from the consumer had reached the subscriber.

Redis Streams as the Solution

Redis Streams provide a more robust and feature-rich messaging system. They are essentially append-only logs where producers add messages, and consumers read them.

Redis Streams addresses the issues we encountered with Pub/Sub:

Avoids Duplicate Operations: Even if the same Runner is running on multiple devices, only one instance can read and acknowledge a message, preventing the duplicate operations that occurred with pub/sub.

Message Persistence: Redis Streams persist messages. If the server crashes, operations can be restarted with the same message.

Guaranteed Delivery and Acknowledgement: Redis Streams provides message acknowledgement, assuring the LiveAPI backend that job requests reach the intended subscriber.

Understand Your Usecase
Redis Streams was selected as a solution for LiveAPI due to its message persistence, guaranteed delivery, message acknowledgement, and ability to prevent duplicate operations.

While Pub/Sub is a simpler solution and better suited for real-time messaging, Streams offers more features and guarantees, making it ideal for use cases where data durability and reliability are critical.

Never Give Up on Learning a New Web Framework. This trick will solve a significant part of the problem.

LinceMathew — Wed, 18 Dec 2024 05:50:28 +0000

Simple Trick to Create API Docs for any Open-source Web Projects

LinceMathew ・ Dec 18 '24

#webdev #programming #opensource #ai