Forem: Burhanuddin Ahmed

Audio Noise Reduction Library Using Rust - Testing the AI Coding Capability

Burhanuddin Ahmed — Mon, 05 Jan 2026 10:13:25 +0000

Early of 2026, I want to try the AI capabilities of building app using unknown tech stack.

I am going to start with Rust.

I have never coded in Rust before.

It's interesting to see how far AI can do to build something beyond just CRUD. With all good software spec, CRUD operation can be done easily by AI agent, but how about a bit more complex operation like compression, custom algorithm, or something that requires complex math operation.

In this writing, I am going to journal my experiment building audio noise reducer. It's only a simple noise reducer operation without AI/ML, so never expect that the output will be very clean.

Step

If we are a domain expert then it's good, we can define the step, flow, the algorithm that will be used.
But here, I spent sometime researching using Claude and some paper regarding noise reduction on audio format.

Here's the AI said.

Basic Process
1. Analysis Phase
The algorithm first analyzes the audio to understand what constitutes "noise" versus "signal." This often happens during silent passages or a dedicated noise profile section where only background noise is present. The algorithm builds a spectral fingerprint of the noise frequencies.
2. Frequency Domain Conversion
The audio is transformed from time domain (waveform) into frequency domain using techniques like Fast Fourier Transform (FFT). This breaks the sound into its component frequencies, making it easier to identify and isolate noise patterns.
3. Noise Identification
The algorithm compares the current audio spectrum against the noise profile. It identifies which frequencies match the noise characteristics and which contain the desired signal. Common noise patterns include hiss, hum, or environmental sounds with consistent spectral signatures.
4. Spectral Subtraction
The identified noise frequencies are subtracted or attenuated from the overall signal. The algorithm reduces the amplitude of frequencies matching the noise profile while preserving frequencies containing speech or music.
5. Smoothing and Refinement
To avoid artifacts like musical noise (random twinkling sounds), the algorithm applies smoothing across time and frequency. This might involve gain reduction that varies gradually rather than abruptly.
6. Reconstruction
The processed frequency data is converted back to time domain audio using inverse FFT, producing the cleaned output signal.

Initially, I want to build the whole thing from scratch using Rust, but I think it will waste of time and waste my token. So here're the libraries Claude suggested.

For FFT operations:

rustfft - Fast Fourier Transform implementation
realfft - Optimized for real-valued signals (typical for audio)

For audio I/O:

hound - Reading/writing WAV files
cpal - Real-time audio input/output

For math/signal processing:

ndarray - Multi-dimensional arrays
num-complex - Complex number operations

High-level concept

Read audio and convert to mono, float.

Frame the signal into overlapping windows.

For each frame:

Apply window (Hann/Hamming).
FFT → magnitude and phase.
Estimate/track noise magnitude spectrum.
Subtract noise spectrum with some safety heuristics.
Recombine modified magnitude with original phase.
iFFT to time domain.

Overlap‑add frames to reconstruct the enhanced signal.

Optionally apply simple post‑processing (clipping, normalization).

After that, I can input this as a context and let the AI creates a plan and the implementation.

Definitely, I wont be one-shot, it will include multiple feedback and more alignment from the user.

Model and Agent

Everything was built using Claude Code with GLM-4.7 as custom model.

Total cost to build this App

Total cost:            $18.41 (costs may be inaccurate due to usage of unknown models)
     Total duration (API):  1h 27m 6s
     Total duration (wall): 5h 42m 37s
     Total code changes:    4954 lines added, 1987 lines removed
     Usage by model:
              glm-4.5-air:  27.5k input, 6.0k output, 86.2k cache read, 0 cache write ($0.1981)
                  glm-4.7:  1.3m input, 166.2k output, 39.1m cache read, 0 cache write ($18.22)

Repo

Check out my experiment here

https://github.com/burhanahmeed/bg-noise-reduction

System Design: Jitter Time for Highly Concurrent API

Burhanuddin Ahmed — Tue, 30 Sep 2025 00:27:26 +0000

Before going deep to the idea, understanding of the jargon is required.

Jitter time is kind of random delay time between request is created and is arrived to the other resources.

Suppose there are two resources interacting from A to B, hundreds or thousands of requests is hitting resource B. Without adding a jitter time, potentially it could hit 100+ request per second (RPS). There's no real number collected but conceptually adding a jitter time would reduce the RPS hitting the resource.

For example, jitter time of 1 minute was added to resource A when calling B. So the request coming from A will be randomized up to 1 minute time, first request can run in the 7th second, another can run on 59th second, another can run on the 32nd second, and so on.

There will always a potential that a request can have the same jitter time, but at least it reduces the potential of all of the requests running at the same time.

Things to consider

Not all of the use cases is fit implemented a jitter time.

Time sensitive action might not be able to be implemented this method.

Example

For a scheduled action or a cron job for example, this will be great place to implement jitter time. So, once the job is executed and bring a lot of request, it will be jittered.

Cron job usually will took some data from database then being run simultaneously.

Example invoice invalidation

Cronjob run
get expired invoices, suppose there are 1000 invoices.
invalidate 1000 invoice. <- before going to this resource, jitter time can be added, so invalidate invoice won't be too heavy.

Comment below what other use case or solution can be done and how does it help you?

Connect on linkedin!

Rate Limit Handling: Using Randomized Delays with Cloud Tasks

Burhanuddin Ahmed — Tue, 29 Apr 2025 11:06:44 +0000

As a developer, we often integrate with external API. There are a lot of consideration when working with external service such as API availability, latency, network failure, API timeout, and rate limit.

We gonna talk about rate limit.

It's a common design to handle and manage system load when exposing the API to external partner. From API's POV, we need to determine like how many request per second our API should handle, otherwise we will reject it to maintain the API latency and other performance metrics.

If we talk from consumer POV, we will thing how our outbound request to the API pass the rate limit or how not to get caught by rate limiter. In low traffic application, it's not a problem, we wont hit the rate limit, but for mid - large scale traffic it might be an issue.

Use case

In the use case, we receive a request from other consumer, it can be direct API call or push pubsub. Where from one request it can generate 10 to 20 API call to external service which is rate limited for 10req/s.

Think about a feature that will allow user to update goods information to other external API provider. User can have many goods and external provider only provide single update not bulk update.

API A is responsible for receiving request from internal consumer and filtering the data (for example, check if user is exist or how many goods does user have). After that, it will get all of the goods data, for example we find 30 datas which will be updated to API B simultaneously.

Let see API B, this is responsible for calling external API. By using API B we can control the request delivery to external API provider.

Above we get 30 datas, means we will get 30 requests almost at the same seconds, maybe only have milis difference.

Since the max RSP is 10, If we don't have any logic to handle rate limit here, it will eventually fail.

Before that, we can add a queue to handle the request delivery to external API. We can use BullMQ.

https://bullmq.io/

BullMQ will help use to manage the rate limit for our outbound request to external API.

import { Queue, Worker, QueueScheduler } from 'bullmq';

// Create a queue with rate limiting options
const queue = new Queue('rate-limited-queue', {
  connection: {
    host: 'localhost',
    port: 6379
  },
  defaultJobOptions: {
    // Limit to 10 jobs per 30 seconds
    limiter: {
      max: 10,
      duration: 30000
    }
  }
});

// Queue Scheduler is required for rate limiting
const scheduler = new QueueScheduler('rate-limited-queue', {
  connection: {
    host: 'localhost',
    port: 6379
  }
});

// Create worker with rate limiting
const worker = new Worker('rate-limited-queue', async (job) => {
  console.log(`Processing job ${job.id} at ${new Date().toISOString()}`);
  await processJob(job.data);
}, {
  connection: {
    host: 'localhost',
    port: 6379
  },
  // Process max 2 jobs concurrently
  concurrency: 2,
  // Rate limit the worker itself
  limiter: {
    max: 5,
    duration: 1000 // 5 jobs per second
  }
});

After implementing BullMQ, if the API does not throw rate limit error, so you can use this solution and don't make everything more complicated.

In our case, we still got rate limit error quite often because the external API latency is sometimes exceeding 1s. That means, all 10 requests must have below 1s latency, for example if 7 requests have < 1s and else have more than 1s, the next 10 coming requests will be executed in the next second, we might see 3 requests fail, because 3 requests from the previous second were not done yet.

Disclaimer, it might be depends on how external API provider handle their rate limit logic.

Before go to more complicated solution using Cloud Task, we can leverage setTimeout() in NodeJS. We can create randomized request to API B using setTimeout.

const delay = Math.floor(Math.random() * 30000); // Random delay 0-30000ms (30 seconds)

setTimeout(async () => {
    // function to call API B here
}, delay);

It's more than enough, if we need something more persistent for distributed system, we can use Cloud Task to create delayed task. It will have retry mechanism automatically when something happen for example a network issue. It will also avoid data loss when the server/pod is shutdown/restarted suddenly.

Maybe you wonder why should GCP Cloud Task?

Our existing cloud solution is GCP, so using Cloud Task will be straightforward for us.

const { CloudTasksClient } = require('@google-cloud/tasks');
const client = new CloudTasksClient();

async function createDelayedTask(payload) {
  const project = 'your-project-id';
  const queue = 'your-queue-name';
  const location = 'us-central1';

  const parent = client.queuePath(project, location, queue);

  // Generate random delay between 0-30 seconds
  const delaySeconds = Math.floor(Math.random() * 31);
  const delayMs = delaySeconds * 1000;

  // Calculate schedule time
  const scheduleTime = new Date();
  scheduleTime.setMilliseconds(scheduleTime.getMilliseconds() + delayMs);

  const task = {
    httpRequest: {
      httpMethod: 'POST',
      url: 'https://your-service.com/process-task-API-B', // API B
      headers: {
        'Content-Type': 'application/json',
      },
      body: Buffer.from(JSON.stringify(payload)).toString('base64'),
    },
    scheduleTime: {
      seconds: scheduleTime.getTime() / 1000,
    },
  };

  try {
    const [response] = await client.createTask({ parent, task });
    console.log(`Task created: ${response.name}`);
    console.log(`Scheduled with ${delaySeconds} seconds delay`);
    return response;
  } catch (error) {
    console.error('Error creating task:', error);
    throw error;
  }
}

While not entirely eliminating the potential for more than 10 simultaneous external API requests, distributing these 30 calls across separate seconds greatly reduces this likelihood.

30 requests from the example in above use case.

Thanks for reading, hope you enjoy it.

Let's connect with me. https://linkedin.com/in/burhanahmeed

Building a Multi-Source AI Agent: Bridging Databases, APIs, and AI Models

Burhanuddin Ahmed — Thu, 13 Feb 2025 03:07:47 +0000

We are experimenting to build a POC to generate chart raw data automatically from a datasource without framework like Langchain, Langgraph, etc. Under the hood, it's just a simple workflow that will involve some processes like inserting the raw data, meta data, fetching both of raw and meta data, interacting with LLM APIs, and showing the response.

We are using Gemini (Vertex AI) for the LLM and Python (FastAPI) as the main weapon.

what is multi-source AI agent?

An AI agent is a system that can do some process to perform one or multiple task for human with less human intervention.

While multi-source here will refer to the entity that will be involved in my workflow process, whether it's an insertion or fetch process.

Here, the entities that will be used in my experiment.

MySQL for storing metadata
MySQL for storing raw data
Vertex AI
Our own FastAPI's APIs

The Architecture

Let's assume, We have two separated workflows, uploader API is for uploading raw data such as CSV file while fetcher API is for getting the raw data for chart which already customized based on AI suggested insight.

Showing data into a chart needs to have predefined purpose, impossible to show all random raw data to a chart without any purpose of what insight we might get.

Insight means like top sales in a month, number of GMV per month in a year, etc. Those thing will be determined by AI agent.

Uploader API

This API is responsible to interact with LLM API and two databases to ingest the raw data into an internal data source. This approach might have a drawback like cost, table and databases maintenance but loading the external data into my own database will make me easier when dealing with huge data and easier to query the data using SQL.

Another alternative besides using MySQL is using Columnar DB, NoSQL or using memory database (but I need the data to be persisted). Each choice will have its own drawback.

Basically, after uploading a CSV, it will create own table in MySQL for each CSV. The system will fist determine the data schema of the file and use it as the reference to create a table.

Each table created will have different schema based on the uploaded CSV file. So We can filter the data and column from database level.

- table_csv_1 
- table_csv_2
- table_csv_3
etc

Another options is by storing the whole CSV data in a column as a raw data or as a JSON data. We didn't see this option as a good approach as it will be difficult to maintain the data particularly when dealing with large datasets, because data processing like filtering the data and column will be occurred in the application logic level.

|ID|entity_name       |raw_data     |created_at
|1 |the name of entity|{...raw data}|2024-11-12

In short, the processes are,

Upload a file
Get a few rows for the sample of the data
throw into LLM to determine the schema for this CSV
save the schema in MySQL for meta data, later We can use it for LLM prompt
perform CREATE TABLE based on the schema
insert the data into newly created table

Fetcher API

If uploader is used for a CSV data upload, fetcher will responsible for generating the data that will be ready to be used by Frontend chart.

The process will involve a process like determine the insight based on the dataset schema and generate the query based for the insight.

Process:

fetch meta data based on entity ID, here we will get the data such as schema, table name, and other meta data.
call LLM API with data schema included as the prompt
the LLM will give us the query that we can use it to query into our raw data

result and challenge

At the end, fetcher API will respond with data such as this for example:

{
  "data": [
    {
      "date": "2024-01-05",
      "total_sales": 10000
    },
    {
      "date": "2024-01-10",
      "total_sales": 90000
    },
    {
      "date": "2024-01-17",
      "total_sales": 92000
    }
  ],
  "insight": "sales per day",
  "summary": "Lorem imsum, lorem ipsum"
}

Later you can use those response to generate chart using chart library you like.

AI can hallucinate and can provide incorrect information. Misleading data schema or SQL query usually are the most common things that LLM response provides incorrectly. It often causes a runtime error because of column not found or wrong data format.

Hello, My name is Burhan. A software engineer who love thinkering about product building, software performance, and islamic finance.

Let's connect on Linkedin https://www.linkedin.com/in/burhanahmeed/

The Right Way to Do HMAC Authentication in ExpressJS

Burhanuddin Ahmed — Wed, 05 Jun 2024 14:30:36 +0000

Authentication is the most important thing when building an API. There are many method to authenticate to an API. Some provider choose to only bearer token, some choose to use a pair of username and password, and many other way to authenticate an API. Besides those, there is another way to authenticate an API, yes, it is using an hmac comparison.

Usually an API webhook use this kind of authentication.

A webhook will be sent by third party system to invoke an API in our system when a certain action happened. In example, when a payment just done by customers or an order just created, etc. We only need to provide a POST API that will do some processes when there is an invocation. What we need to do is providing an API then usually in the third party system there is a configuration to set the endpoint to hit when an action is triggered.

example: payment just done then send payment status notification to our system via webhook, so we can change the status in our DB.

One of the popular authentication method is using HMAC hash, where third party system will send a hash code in the header and our API will authenticate from the hash code by generating HMAC then compare the HMAC hash code result with the hash code sent by partner.

The idea of this authentication is to make sure that the only party can invoke our webhook API is only the particular partner we have been configured, so not everyone can hit our API.

Example We want only Stripe that can invoke the webhook to send payment status. We dont want some random guy hitting and spamming the webhook using Postman and send random data.

At this point, we should follow the authentication method that is provided by the third party system.

Generate the Hmac

The most common way to generate a hmac is that the third party system will give us a secret code to generate hmac and then use the request body as encoding string (Buffer is also possible). Usually partner will have a documentation about how they generate the hmac hash and we just need to follow the logic using our prefered programming language.

func generateHMAC(message string, secretKey string) (string, error) {
    key := []byte(secretKey)
    h := hmac.New(sha256.New, key)

    _, err := h.Write([]byte(message))
    if err !=  nil {
        return "", err
    }

    // Get the HMAC value
    signature := h.Sum(nil)

    // Encode the signature in hexadecimal format
    encodedSignature := hex.EncodeToString(signature)

    return encodedSignature, nil
}

or in Javascript

function generateHmac(
  message: string,
  secretKey: string,
  options?: HmacOptions
): string {
  const algorithm = options?.algorithm || 'sha256';
  const encoding = options?.encoding || 'hex';

  const hmac = crypto.createHmac(algorithm, secretKey);
  hmac.update(message);
  const signature = hmac.digest(encoding);

  return signature;
}

Above just a simpel way to generate an HMAC hash.

How to do it the right way

As the title said, there is nothing wrong about the function to generate HMAC. The problem is in the argument that were sent to the function. I have this use case when using ExpressJS. Are we supposed to send the correct message to generateHmac() or not?

Usually we will use this method, pass req.body in the argument. Basically it will work fine until the partner send an encoded character, it will start failing.

import express from 'express';

const app: express.Application = express();

// Configure body-parser to handle JSON and URL-encoded data
app.use(express.json());
app.use(express.urlencoded({ extended: false }));

// Define a simple route that responds with a message
app.get('/', (req: express.Request, res: express.Response) => {
  const hash = generateHmac(JSON.stringify(req.body), process.env.SECRET_CODE);
  if (hash !== req.headers.hash) { // throw an error here }

  res.send('Hello from a simple Express.js app!');
});

express.json() will automatically decode the character into a readable one. Thats why the message argument in our end and in partner end to generate the HMAC would be different.

I give you an example. The partner send the message in the payload Hi, Tom \u0026 Jerry, bodyParser will translate it as Hi, Tom & Jerry and we will generate HMAC using the translated string, which will generate different HMAC code to partner end.

Solution

Using raw body will always be the workaround. We can get raw data from express.json(), then append it into the request object.

app.use(
  express.json({
    verify(req: express.Request, res, buf, encoding) {
      req.rawBuffer = buf;
    }
  })
);

. . . 
. . .

app.get('/', (req: express.Request, res: express.Response) => {
  const hash = generateHmac(req.rawBuffer, process.env.SECRET_CODE);
  if (hash !== req.headers.hash) { // throw an error here }

  res.send('Hello from a simple Express.js app!');
});

Make sure generateHmac() parameters should accept Buffer too.

By using raw data, we can make sure that the message we use to generate HMAC is same between partner end and our end, there is no data discrepancy.

When testing using Postman or any other API client, you can use online hmac generator to generate the hmac token then pass it in the header.

Thanks for reading.

👋🏻 I'm Burhan, let's connect https://linkedin.com/in/burhanahmeed

Getting Started with Load Testing using Locust

Burhanuddin Ahmed — Fri, 29 Mar 2024 06:55:06 +0000

What is Locust

Locust is a load testing tool written in Python. The test script is also written in Python. To use this tools, you need at least very basic knowledge of Python.

https://locust.io/

Across many testing framework, Locust is quite simple and having pretty good and simple UI. You only need to install Python in your machine and just install the tools.

How to install it

Installing Locust is quite simple, just need pip and you can run it.

pip install locust

I usually using virtual env before installing the tools.

Create a virtual env

python3 -m venv venv

Activate the venv

source venv/bin/activate

Then you can run Locust here

Create the testing script

This one simple example to load test on login API.

# index.py

from locust import HttpUser, task, between

class LoginUser(HttpUser):
    wait_time = between(2, 5)  # Wait time between simulated users

    @task
    def login(self):
        # Define the payload for the login request
        login_payload = {
            'username': 'your_username',
            'password': 'your_password'
        }

        # Send a POST request to the login endpoint
        response = self.client.post('/login', json=login_payload)

        # Check if the login was successful
        if response.status_code == 200:
            print("Login successful")
        else:
            print("Login failed")

Now let say, your API needs a token before you hit it, this how we will do it.

from locust import HttpUser, task, between

class UserApiTest(HttpUser):
    wait_time = between(2, 5)  # Wait time between simulated users

    def on_start(self):
        # Log in and get the authentication token
        login_payload = {
            'username': 'your_username',
            'password': 'your_password'
        }
        response = self.client.post('/login', json=login_payload)
        if response.status_code == 200:
            self.auth_token = response.json().get('token')
        else:
            print("Login failed")

    @task
    def get_users(self):
        # Send a GET request to the /users endpoint with the authentication token
        headers = {'Authorization': f'Bearer {self.auth_token}'}
        response = self.client.get('/users', headers=headers)

        # Check if the request was successful
        if response.status_code == 200:
            print("Get users successful")
        else:
            print("Get users failed")

Run it!

To run it, it's very simple.

locust -f index.py

Defining Custom Many-to-many Relationship in NestJS TypeORM.

Burhanuddin Ahmed — Sun, 31 Dec 2023 03:23:30 +0000

If you are working using NestJS and using TypeORM for the database ORM, you might encounter some issues when trying to create table relationship.

I understand you, the documentation does not seem clear for some people and ChatGPT can't event help us.

This post will help you to create many-to-many relationship on custom pivot table using TypeORM. There are some decorators on TypeORM.

@OneToMany() 
@ManyToOne() 
@ManyToMany() 
@OneToOne()

Here, we will create many to many entities, we will use OneToMany and ManyToOne decorators instead of ManyToMany. We can use ManyToMany if we want our table is created and synchronized automatically and don't have custom field, but here I create my own DDL migration script to create my tables because I want to have custom field on my pivot table.

I recommend you to not sync the db automatically and create your own db migration.

Before we see the table and pivot table, I will share the DDL I use to create the table.

table 1

CREATE TABLE `user` (
  `id` int NOT NULL AUTO_INCREMENT,
  `email` varchar(255) NOT NULL,
  `fullname` varchar(255) DEFAULT NULL,
  `password` varchar(255) DEFAULT NULL,
  `firebase_uid` varchar(255) DEFAULT NULL,
  `created_at` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
  `updated_at` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  PRIMARY KEY (`id`),
  KEY `IDX_user_email` (`email`)
) ENGINE=InnoDB AUTO_INCREMENT=5 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci;

table 2

CREATE TABLE `social_account` (
  `id` int NOT NULL AUTO_INCREMENT,
  `source` varchar(255) NOT NULL,
  `username` varchar(255) NOT NULL,
  `fullname` varchar(255) DEFAULT NULL,
  `image` text,
  `description` varchar(255) DEFAULT NULL,
  `external_id` varchar(255) NOT NULL,
  `meta` text,
  `last_crawl_at` timestamp NOT NULL,
  `created_at` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
  `updated_at` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  PRIMARY KEY (`id`),
  KEY `IDX_social_account_username` (`username`),
  KEY `IDX_social_account_external_id` (`external_id`)
) ENGINE=InnoDB AUTO_INCREMENT=4 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci;

pivot table

CREATE TABLE `subscriptions` (
  `user_id` int NOT NULL,
  `social_account_id` int NOT NULL,
  `created_at` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
  `updated_at` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
  PRIMARY KEY (`user_id`,`social_account_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci;

We can see, in pivot table we are having additional fields for created_at and updated_at.

So basically the relation is as follows,

user -> subscription -> social_account

A user can subscribe to many social account, while a social account can be subscribed by many users.

From here, now we can go to NestJS entity to define the relationship.

User

// user.entity.ts
import {
  Entity,
  PrimaryGeneratedColumn,
  Column,
  OneToMany,
} from 'typeorm';
import { SocialAccount } from '../social-accounts/social-account.entity';
import { Subscription } from '../subscriptions/subscription.entity';

@Entity()
export class User {
  @PrimaryGeneratedColumn()
  id: number;

  @Column()
  email: string;

  @Column({ nullable: true })
  fullname: string;

  @Column({ nullable: true })
  password: string;

  @Column({ nullable: true })
  firebase_uid: string;

  @Column({
    type: 'timestamp',
    default: () => 'CURRENT_TIMESTAMP',
    nullable: false,
  })
  public created_at: Date;

  @Column({
    type: 'timestamp',
    default: () => 'CURRENT_TIMESTAMP',
    onUpdate: 'CURRENT_TIMESTAMP',
  })
  public updated_at: Date;

  @OneToMany(() => Subscription, (subs) => subs.user)
  public socialAccounts: SocialAccount[];
}

social account

// social-account.entity.ts
import {
  Entity,
  PrimaryGeneratedColumn,
  Column,
  OneToMany
} from 'typeorm';
import { User } from '../user-managements/user.entity';
import { Subscription } from '../subscriptions/subscription.entity';

@Entity()
export class SocialAccount {
  @PrimaryGeneratedColumn()
  id: number;

  @Column()
  source: string;

  @Column()
  username: string;

  @Column({ nullable: true })
  fullname: string;

  @Column({ nullable: true, type: 'text' })
  image: string;

  @Column({ nullable: true })
  description: string;

  @Column()
  external_id: string;

  @Column({ nullable: true, type: 'text' })
  meta: string;

  @Column({ type: 'timestamp', nullable: false })
  last_crawl_at: Date;

  @Column({
    type: 'timestamp',
    default: () => 'CURRENT_TIMESTAMP',
    nullable: false,
  })
  public created_at: Date;

  @Column({
    type: 'timestamp',
    default: () => 'CURRENT_TIMESTAMP',
    onUpdate: 'CURRENT_TIMESTAMP',
  })
  public updated_at: Date;

  @OneToMany(() => Subscription, (subs) => subs.socialAccount)
  public user: User[];
}

subscription

// subscription.entity.ts
import {
  Entity,
  PrimaryColumn,
  Column,
  ManyToOne,
  JoinColumn,
} from 'typeorm';
import { User } from '../user-managements/user.entity';
import { SocialAccount } from '../social-accounts/social-account.entity';

@Entity({ name: 'subscriptions' })
export class Subscription {
  @PrimaryColumn()
  user_id: number;

  @PrimaryColumn()
  social_account_id: number;

  @Column({ type: 'timestamp', default: () => 'CURRENT_TIMESTAMP' })
  public created_at: Date;

  @Column({
    type: 'timestamp',
    default: () => 'CURRENT_TIMESTAMP',
    onUpdate: 'CURRENT_TIMESTAMP',
  })
  public updated_at: Date;

  @ManyToOne(() => User, (user) => user.socialAccounts)
  @JoinColumn({ name: 'user_id' })
  user: User;

  @ManyToOne(() => SocialAccount, (socialAccount) => socialAccount.user)
  @JoinColumn({ name: 'social_account_id' })
  socialAccount: SocialAccount;
}

We can see we are using OneToMany in table 1 and table 2, while there are two ManyToOne in pivot table.

You can copy the concept as above if you have a problem defining many-to-many relationship with custom table.

See you!!!

A CRUD ExpressJS example, with React and Vue example usage

Burhanuddin Ahmed — Mon, 30 Oct 2023 14:47:52 +0000

This post might be "just another ExpressJS example", but I hope it will be useful enough for the beginner to know how to create a RESTful API in ExpressJS. I am not only providing a basic project example with awful structure.

I create the repository to give you the example ExpressJS and the usage of the API in ReactJS.

Here, you will learn by the project example.

I have structured the project using a simple Model-Service-Controller pattern. It's simple pattern but powerful enough to get started and working on your client project right away.

What you can learn

CRUD
User authentication
File upload
How to use it in ReactJS
How to use it in VueJS (Soon)
Sequelize

Here is the repository, dont forget to share and star it.

https://github.com/burhanahmeed/nodejs-react-vue-project-starter

I will update the repo along the way.

😬 I am not writing another CRUD tutorial, you can see the repo and learn from it.

Personal note: Dokerize Elixir App

Burhanuddin Ahmed — Tue, 05 Sep 2023 13:39:19 +0000

After we have completed creating our Elixir CRUD app, now we want to wrap it into a container. We will use Docker which is a popular tool to container the app.

To container it, we need to create a dockerfile in root app.

We can add .dockerignore. It's optional but ok if we have it.

.dockerignore

# Elixir Artifacts
/_build/
/deps/
/doc/
/cover/
/.fetch
*.ez
APPNAME-*.tar
erl_crash.dump


# Node Artifacts
npm-debug.log
/assets/node_modules/
/priv/static/


# File uploads
/uploads
/test/uploads


# Docker only
/test/
/.iex.exs

OK, after we add it now we can add dockerfile.

dockerfile

FROM elixir:1.15.1 as builder
WORKDIR /app
COPY . .
ENV MIX_ENV=prod
COPY lib ./lib
COPY mix.exs .
COPY mix.lock .
COPY config/prod.env.exs config/

RUN mix local.rebar --force \
    && mix local.hex --force \
    && mix deps.get \
    && mix release

EXPOSE 8081

CMD ["_build/prod/rel/todo_app/bin/todo_app", "start"]

If we want more advanced dockerfile, we can use multi-stage build dockerfile. By using multi-stage build we can have lighter image because it will only contain Elixir distribution file.

But in this post, We will only provide single-stage build for more simplicity.

Happy Coding!!! 🍻

Personal note: Build Elixir CRUD without Phoenix using MySQL and MongoDB

Burhanuddin Ahmed — Tue, 01 Aug 2023 13:40:15 +0000

My personal note about learning Elixir.

Finding learning resources for Elixir is not as easy as JS learning resources. So I initiate this post to document my journey learning Elixir.

Basically I want to practice to create a create, read, list, update, and delete using MongoDB and MySQL.

Here, I will use Elixir without Phoenix framework, I want to know how Elixir work.

Make sure you have installed Elixir in your system.

To start a new project, use this command mix new <app-name> --sup

mix new todo_app --sup

A --sup option can be given to generate an OTP application skeleton including a supervision tree. Normally an app is generated without a supervisor and without the app callback.

After initialize new project, it will create directory structure like this.

You will see Elixir extension with .ex and .exs. The ".ex" extension is used for Elixir source code files that are compiled into bytecode, while the ".exs" extension is used for Elixir script files that are interpreted directly.

Adding dependency

Go to /mix.exs, inside deps add the dependency.

 defp deps do
    [
      {:plug_cowboy, "~> 2.5"},
      {:jason, "~> 1.3"},
      {:mongodb_driver, "~> 0.8"},
      {:myxql, "~> 0.6.0"}
    ]
  end

After adding the dependency, run mix deps.get.

plug_cowboy is a kind of Erlang web server, since we didn't use any framework like Phoenix, we can use it.
jason is a JSON parser and generator for Elixir.
mongo_driver for establishing connection with MongoDB, and myxql for connecting with MySQL.

Set up the dependency

After installing the dependencies, we need to set it up in our application.ex.

According to Elixir site

Applications are the idiomatic way to package software in Erlang/OTP. To get the idea, they are similar to the "library" concept common in other programming languages, but with some additional characteristics.

An application is a component implementing some specific functionality, with a standardized directory structure, configuration, and life cycle. Applications are loaded, started, and stopped. Each application also has its own environment, which provides a unified API for configuring each application.

Here we will define the entry point of our app. Let's head to lib/todo_app/application.ex

# lib/todo_app/application.ex

defmodule TodoApp.Application do
alias Mix.Tasks.Compile.App

  use Application

  @impl true
  def start(_type, _args) do
    children = [
      {
        Plug.Cowboy,
        scheme: :http,
        plug: TodoApp.Router,
        options: [port: Application.get_env(:todo_app, :port)]
      },
      {
        Mongo,
        name: :mongo,
        url: Application.get_env(:todo_app, :database_url),
      },
      {
        MyXQL,
        name: :myxql,
        hostname: Application.get_env(:todo_app, :mysql_host),
        username: Application.get_env(:todo_app, :mysql_username),
        password: Application.get_env(:todo_app, :mysql_password),
        database: Application.get_env(:todo_app, :mysql_database),
      }
    ]

    opts = [strategy: :one_for_one, name: TodoApp.Supervisor]
    Supervisor.start_link(children, opts)
  end
end

We can see that we have setup for Cowboy, Mongo, and MySQL inside the application.ex. We can also see there are Application.get_env(:todo_app, :mysql_database), it means that we calling the value from an environment variable. Now let's define the env variable.

Env Variable

Go to config/, create config.exs and dev.env.exs.

Put this value into config.exs

# config/config.exs

import Config

# Import environment specific config. This must remain at the bottom
# of this file so it overrides the configuration defined above.
import_config "#{config_env()}.env.exs"

The config file will import the code from dev.env.exs, if we use prod then we must have prod.env.exs.

In dev.env

# config/dev.env.exs

import Config

config :todo_app, port: 8081
config :todo_app, database_url: "mongodb://localhost:27017/sultanads"
config :todo_app, database: "db_demo"
config :todo_app, pool_size: 3
config :todo_app, :basic_auth, username: "abc", password: "abc"
config :todo_app, mysql_host: "localhost"
config :todo_app, mysql_username: "root"
config :todo_app, mysql_password: ""
config :todo_app, mysql_database: "dev_mdom"

Read more about config here https://hexdocs.pm/elixir/main/Config.html

Next, define the router.

Router

Here in router, we will define the endpoint and add little CRUD logic.

https://github.com/burhanahmeed/elixir-mongodb-app/blob/master/lib/todo_app/router.ex

That's all, I can't explain much about it, but as a personal note it's enough for me to get started again one day.

Next, I will try with Phoenix, dockerize, and CI/CD.

Firebase: Applying Feature Flag using Remote Config

Burhanuddin Ahmed — Sun, 18 Dec 2022 13:56:42 +0000

Firebase Remote Config is a powerful tool for app developers that allows them to easily modify the appearance and behavior of their app without requiring users to download an update. This can save time and resources, and make it easier to roll out changes to an app quickly and efficiently.

What is a feature flag?

A feature flag is a process in software development to manage the availability of a feature. Dev people or anyone who have access to the configuration can enable or disable the feature using this flag remotely without deploying the code again.

Let's say, I have a NextJS app and I want to deploy a new button, I realized that the button is not working as I expected. Instead of rolling back to the previous button version, I just can disable the new button flag and the site will show the old button again.

Before we head to the use case, you can look at the very simple getting started code below.

// Initialize Firebase
firebase.initializeApp({
  apiKey: "YOUR_API_KEY",
  authDomain: "YOUR_AUTH_DOMAIN",
  databaseURL: "YOUR_DATABASE_URL",
  projectId: "YOUR_PROJECT_ID",
  storageBucket: "YOUR_STORAGE_BUCKET",
  messagingSenderId: "YOUR_MESSAGING_SENDER_ID"
});

// Get a reference to the Remote Config service
const config = firebase.remoteConfig();

// Set default values
config.defaultConfig = ({
  "welcome_message": "Welcome to our app!"
});

// Fetch the config values
config.fetch()
  .then(() => {
    // Activate the fetched config
    return config.activate();
  })
  .then(() => {
    // Get the welcome message from the config
    const welcomeMessage = config.getValue("welcome_message").val();

    // Update the UI with the welcome message
    document.getElementById("welcome-message").textContent = welcomeMessage;
  })
  .catch((error) => {
    // There was an error fetching the config
    console.error(error);
  });

Just FYI, the code above is the old version of Firebase SDK (version 8), I will tell you the latest code that uses SDK version 9.

You see config.activate(); here is purposed for getting the latest value.

If you don't have activate(), your getValue() will only return an empty value.

Why it matters

The activate method is used to apply the fetched set of parameter values to your app. When you call fetch, the Remote Config API retrieves the latest set of parameter values from the server and stores them locally. These values are not applied to your app until you call activate.

Okay, now I will show you how I do it in NextJS.

What you need to prepare is to have a Firebase App ready and NextJS ready too. I will not tell you how to setup NextJS and Firebase, I expect you to understand about setting up NextJS including the config like dotenv.

Then go to Firebase project settings to see the API and secret key.

Now in NextJS root dir create .env with content like the below.

NEXT_PUBLIC_FIREBASE_APP_ID=
NEXT_PUBLIC_FIREBASE_API_KEY=
NEXT_PUBLIC_FIREBASE_MESSAGE_ID=
NEXT_PUBLIC_FIREBASE_PROJECT_ID=
NEXT_PUBLIC_FIREBASE_GA_ID=

To make it neat, create a global config in /src/config/index.ts

export const firebaseApiKey = process.env.NEXT_PUBLIC_FIREBASE_API_KEY || '';
export const firebaseAppId = process.env.NEXT_PUBLIC_FIREBASE_APP_ID || '';
export const firebaseProjectId =
  process.env.NEXT_PUBLIC_FIREBASE_PROJECT_ID || '';
export const messagingSenderId =
  process.env.NEXT_PUBLIC_FIREBASE_MESSAGE_ID || '';
export const googleAnalyticId = process.env.NEXT_PUBLIC_FIREBASE_GA_ID || '';

The next step is to create Firebase config. Create a file in /src/utils/firebaseConfig.ts.

You can follow my style or use yours also can.

// Import the functions you need from the SDKs you need
import { initializeApp } from 'firebase/app';
import { getRemoteConfig } from 'firebase/remote-config';

import {
  firebaseApiKey,
  firebaseAppId,
  firebaseProjectId,
  messagingSenderId,
  googleAnalyticId,
} from '@/config';
// TODO: Add SDKs for Firebase products that you want to use
// https://firebase.google.com/docs/web/setup#available-libraries

// Your web app's Firebase configuration
const firebaseConfig = {
  apiKey: firebaseApiKey,
  authDomain: `${firebaseProjectId}.firebaseapp.com`,
  projectId: firebaseProjectId,
  storageBucket: `${firebaseProjectId}.appspot.com`,
  messagingSenderId,
  appId: firebaseAppId,
  measurementId: googleAnalyticId,
};

// Initialize Firebase
export const app = initializeApp(firebaseConfig);

let remoteConfigVar;
if (typeof window !== 'undefined') {
  remoteConfigVar = getRemoteConfig(app);
  remoteConfigVar.settings.minimumFetchIntervalMillis = 600000;
}
export const remoteConfig = remoteConfigVar;

I will highlight this line typeof window !== 'undefined'. Since our NextJS is running in the server side, Remote Config won't be able to be initialized so we need to check if window is not undefined.

Also, we have this remoteConfigVar.settings.minimumFetchIntervalMillis = 600000;. By default, Firebase will cache the config for 12 hours, so any changes won't be applied before 12 hours, but we can add a minimum fetch interval as low as 10 minutes.

Then in NextJS, let's say in the index.ts page.

import { getValue, fetchAndActivate } from 'firebase/remote-config';
import { remoteConfig } from '@/utils/firebaseConfig';

Then

  const [showBtn, setShowBtn] = useState(false);

  const getConfigValue = () => {
    const firebaseRemoteConfig = remoteConfig;
    if (firebaseRemoteConfig) {
      fetchAndActivate(firebaseRemoteConfig).then(() => {
        setShowBtn(getValue(firebaseRemoteConfig, 'show_button').asBoolean());
    }
  };

  useEffect(() => {
    getConfigValue();
  }, []);

And in JSX,

<div className="p-16 text-white">
  {showBtn ? (
    <Button type="primary">New Button</Button>
   ) : (
    <Button type="primary">Old Button</Button>
   )}
</div>

Don't forget to have the config set in Firebase dashboard.

Next, I will try to cover up another Firebase topic, see ya!

Chapter 3: Dokerize Laravel App with Nginx and Postgres.

Burhanuddin Ahmed — Wed, 23 Nov 2022 14:51:41 +0000

Postgres SQL

It's time to set up our database in Docker. Because the app is relatively small and built on monolith architecture, so we will install it inside Docker.

Now we can go to our /infra/pgsql and add a new folder named script and then create initdb.sh inside it.

Initdb.sh will be used to create a script to create PG user and PG databases. In this case, my app is using two databases within the same DB instace.

So we first, create a function to create the user, database, and privilege. Then we can loop it over the number of databases we want to create.

initdb.sh

#!/bin/bash

set -e
set -u

function create_user_and_db() {
  local database=$1

    echo "  Creating user and database '$database'"
    psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" <<-EOSQL
        CREATE USER $database;
        CREATE DATABASE $database;
        GRANT ALL PRIVILEGES ON DATABASE $database TO $database;
EOSQL
}

if [ -n "$POSTGRES_MULTIPLE_DATABASES" ]; then
    echo "Multiple database creation requested: $POSTGRES_MULTIPLE_DATABASES"
    for db in $(echo $POSTGRES_MULTIPLE_DATABASES | tr ',' ' '); do
        create_user_and_db $db
    done
    echo "Multiple databases created"
fi

If you want to see a different way to install PG in Docker, you can check here.

Inside pgsql folder, we can create a Dockerfile

FROM postgres:14.1
COPY script/initdb.sh /docker-entrypoint-initdb.d/

The initdb.sh will be copied to docker-entrypoint-initdb.d, so it will automatically run when we run docker-compose.

So after we have added all of those items, here is the complete docker-compose.yml

version: "3.7"

networks:
  app-network:
    driver: bridge

services: 
  app:
    container_name: app
    build:
      context: ./infra/app
      dockerfile: Dockerfile
    image: php-laravel-7.0
    restart: unless-stopped
    tty: true
    working_dir: /var/www
    volumes: 
      - ./:/var/www
    networks: 
      - app-network

  db:
    container_name: pgsql
    build:
      context: ./infra/pgsql
      dockerfile: Dockerfile
    image: pgsql-docker
    restart: unless-stopped
    expose:
     - "5432"
    ports:
      - "5433:5432"
    volumes: 
      - ./infra/pgsql/script:/docker-entrypoint-initdb.d
      - ./infra/pgsql/data:/var/lib/postgresql/data
    environment:
      - POSTGRES_USER=${DB_USERNAME}
      - POSTGRES_MULTIPLE_DATABASES=main_db,logger_db // you change this
    networks:
      - app-network

  nginx:
    image: nginx:1.19.8-alpine
    container_name: nginx
    restart: unless-stopped
    tty: true
    ports: 
        - 8100:80
    volumes: 
        - ./:/var/www
        - ./infra/nginx/conf:/etc/nginx/conf.d
    networks: 
        - app-network

Now we can run docker-compose build then docker-compose up -d, if we don't see any error, so we can assume our docker set up is a success.

We are still not finished yet because we did manual DB migration, yes we didn't use Laravel migration.

We need to dump the DB manually like this. You can adjust the db dump path.

docker-compose exec -T db pg_restore -U postgres -d main_db < db_dump/dump-<DB NAME>-202203201444.sql

Because we are using
- ./infra/pgsql/data:/var/lib/postgresql/data in our docker-compose, so we expect our database data to be persisted even if the docker is shut down.

One more thing, we need to run these two script to make our Laravel app working.

docker-compose exec -T app php artisan key:generate

docker-compose exec -T app composer install

If we want to run laravel db migration instead, we can run this command.

docker-compose exec -T app php artisan migrate

That's all from my post.