Forem: João Paulo

CAP Theorem: The Choice Between Consistency and Availability in Distributed Systems

João Paulo — Sun, 22 Sep 2024 14:32:29 +0000

The CAP Theorem is a concept in distributed systems that helps us understand the trade-offs between Consistency, Availability and Partition Tolerance.

Consistency means that all clients see the same data, regardless of which instance of the application they are connected to. In other words, the information is the same across all points in the system.

Availability refers to the system's ability to respond to user requests at all times. This means that even if something goes wrong, the system should remain operational and accessible.

Partition Tolerance is the ability of the system to continue functioning even when network failures occur that prevent communication between nodes.

When a partition problem occurs in the network, the nodes in the distributed system may become unable to communicate. In these cases, we have to make a choice: if we prioritize consistency, the system may lose availability; if we prioritize availability, the nodes may operate inconsistently until the partition is resolved.

The CAP Theorem helps us understand, at a high level, the limitations of distributed systems, especially regarding consistency, availability, and partition tolerance. It's important to remember that the theorem simplifies the real complexity of systems and the difficulty of achieving an ideal balance among these three properties in practical scenarios.

Understanding the CAP Theorem is crucial for software architects and developers, as it forces us to make informed decisions about how to design our systems. Ultimately, each choice will have a direct impact on the user experience and the overall effectiveness of the system.

Idempotency Explained: Ensuring Reliable API Calls. A practical example in Nestjs

João Paulo — Sat, 14 Sep 2024 22:42:29 +0000

Idempotency refers to an operation that can be performed multiple times without causing additional effects. In other words, no matter how many times the same action is repeated, the result will always be the same as if it had been executed only once.

Idempotency ensures that the operation is only processed once, even if it's executed multiple times, preventing duplicate payments or unintended outcomes. This is particularly useful in scenarios where operations, such as processing a credit transaction, might be retried due to network failures or system crashes.

In the context of web development, certain HTTP methods are idempotent and others no.

GET: Retrieving a resource using a GET request should not change the state of the resource on the server. Whether you request the same resource once or multiple times, the result should be the same.
PUT: Updating a resource using a PUT request should result in the same resource state regardless of how many times the request is made with the same data.

Some HTTP methods are not idempotent, meaning that repeated applications of the same request can lead to different results. Two common examples are POST and PATCH:

POST: The POST method is typically used to create new resources or submit data. Each POST request can create a new resource or initiate a new action, so sending the same POST request multiple times can result in multiple new resources or actions being created.
PATCH: The PATCH method is used to apply partial modifications to a resource. Since the PATCH request updates only certain fields, the outcome can change with each request, depending on the state of the resource before the update.

Why Idempotency Matters

Idempotency is crucial in systems where reliability and consistency are important. It helps in:

Fault Tolerance: Idempotent operations ensure that retrying operations won’t lead to inconsistent states or unintended effects.
Simplicity: Designing systems with idempotent operations can simplify error handling and recovery processes.
Concurrency: Idempotency helps in managing concurrent operations, reducing the risk of conflicting changes.

Implementing an idempotent endpoint in Nestjs

1. Create the Idempotency Middleware



import {
  Injectable,
  NestMiddleware,
  BadRequestException,
} from '@nestjs/common';
import { Request, Response, NextFunction } from 'express';
import { IdempotencyService } from '../services/idempotency.service';

@Injectable()
export class IdempotencyMiddleware implements NestMiddleware {
  constructor(private readonly idempotencyService: IdempotencyService) {}

  async use(req: Request, res: Response, next: NextFunction) {
    const idempotencyKey = req.headers['x-idempotency-key'] as string;
    if (!idempotencyKey) {
      throw new BadRequestException('Idempotency key is missing');
    }

    const cachedKey = await this.idempotencyService.getKey(idempotencyKey);

    if (cachedKey) {
      res.status(200).json(cachedKey);
      return;
    }

    res.on('finish', async () => {
      if (res.statusCode < 400) {
        await this.idempotencyService.saveKey(idempotencyKey);
      }
    });
    next();
  }
}

2. Create the Idempotency Service



import { Inject, Injectable } from '@nestjs/common';
import {
  iDistributedCacheService,
  IDistributedCacheService,
} from './distributed-cache.service';
import { TEN_MINUTES } from '../constants/app.constants';

@Injectable()
export class IdempotencyService {
  constructor(
    @Inject(iDistributedCacheService)
    private readonly distributedCache: IDistributedCacheService,
  ) {}

  async getKey(key: string): Promise<string> {
    return await this.distributedCache.get(key);
  }

  async saveKey(key: string): Promise<void> {
    this.distributedCache.set(key, key, TEN_MINUTES);
  }
}

Note, in this case i used a general cache service, you can store it in a simple HashMap local or something more robust like redis

3. Register the middleware in your app module



import { MiddlewareConsumer, Module, RequestMethod } from '@nestjs/common';
import { TransactionsModule } from './modules/transactions/transactions.module';
import { PrismaModule } from './modules/prisma/prisma.module';
import { PrismaService } from './modules/prisma/prisma.service';
import { ClientsModule } from './modules/clients/clients.module';
import { ConfigModule } from '@nestjs/config';
import { IdempotencyMiddleware } from './libs/commons/middlewares/idempotency.middleware';
import { RedisDistributedCacheService } from './libs/commons/services/redis-distributed-cache.service';
import { iDistributedCacheService } from './libs/commons/services/distributed-cache.service';
import { IdempotencyService } from './libs/commons/services/idempotency.service';

@Module({
  imports: [
  ],
  providers: [
    IdempotencyService
  ],
})
export class AppModule {
  configure(consumer: MiddlewareConsumer) {
    consumer
      .apply(IdempotencyMiddleware)
      .forRoutes({ path: 'transactions', method: RequestMethod.POST });
  }
}

Yow can now test the implemetation. Make a POST request to the route that you registered with the x-idempotency-key in header. The server should not proccess duplicate requests.

Thank you for reaching this point. If you need to contact me, here is my email: joaopauloj405@gmail.com

Must know concept for Backend Developers: Database transactions

João Paulo — Mon, 02 Sep 2024 13:23:53 +0000

Imagine you want to transfer money to your mother. You will need to withdraw money from your account and send it to hers.

You withdraw the money from your account, but when you go to make the transfer, the operation fails. You check your account and the money is no longer there, but it also hasn't reached your mother's account.

How can you prevent this from happening? Database transactions

What is?

A database transaction is a sequence of operations performed within a database management system as a single logical unit of work.

Transactions are closely related to database integrity and ensure the consistent state of a database even in the event of failure.

How it works?

The flow is simple and usually adopts an all or nothing policy:

Begin the transaction
Execute the operations (data manipulations, queries, etc)
If no errors, commit the transaction, if an error occurs, rollback.

Typescript Example:



  async create(transaction: Transaction, payable: Payable): Promise<Transaction> {
    return this.prisma.$transaction(async (prisma) => {
      const createdTransaction = await prisma.transaction.create({
        data: {
          ...transaction,
        },
      });

      await prisma.payable.create({
        data: {
          ...payable,
        },
      });

      return createdTransaction;
    });
  }

How the event loop enables asynchronous programming

João Paulo — Sun, 14 Jul 2024 19:48:03 +0000

Javascript is a single-threaded programming language (only runs one thread at a time), and runs its code sequentially. With the event loop we can execute tasks in more threads. Understanding this component allows us to create more efficient and performant programs.

How does event loop work?

Before talking about the event loop, it is necessary to talk about the call stack.

The call stack is the structure that allows the ordering and execution of our program. It is also the one that registers function calls.

As said, Javascript runs the code step by step sequentially. First, it allocates memory for variables and then executes the functions at the top of the stack. This flow has a problem: if a function takes a long time to respond, it will block other functions from performing their tasks until it responds.

In real applications, these functions are typically linked to api calls.

To solve this problem we use Javascript Web Apis. With this, we have access to non-blocking functions that run on their own threads and not on the main one and thus do not block others from executing their tasks.

The event loop is what allows JavaScript to perform asynchronous operations. It monitors the call stack and the task queue. The event loop continually checks whether the call stack is empty. If so, it takes the first task from the task queue and moves it to the call stack, where it is executed

Therefore, thanks to the event loop, we can perform operations efficiently and, most importantly, without blocking the execution of the main code.

Thank you for reaching this point. If you need to contact me, here is my email: joaopauloj405@gmail.com

A little about promises in Javascript

João Paulo — Sun, 14 Jul 2024 00:53:51 +0000

JavaScript is a synchronous programming language; however, due to the callback functions, we can make it work as an asynchronous programming language.

The concept of promises are very similar to promises we make in real life, we guarantee that something will be done. A promise can be kept or broken and you hope to know what the promise ended up being.

In terms of code, this is precisely what occurs. Promises can assume the statuses of pending, resolved, or rejected.

The promise object possesses both prototype methods and static methods. The prototypes are those that can only be applied to instances of the promise class. There are three prototypes: then, catch, finally. They all return a promise and handle the states of onFulfilled, onRejected, onFinally, respectively.

There are 4 static methods: reject and resolve, which help you create resolved or rejected promises and all and race. “all” receives an array of promises and returns a promise that resolves when all past promises are resolved or a promise that rejects with the reason for the first promise that was rejected. “race” also receives an array of promises and returns a promise that is resolved or rejected as soon as one of the passed promises is resolved or rejected. All promises run in parallel. This is an interesting feature that allows you to improve the performance of your code.

In modern JavaScript code, you will see a lot of use of async and await, which were introduced to simplify promise management with much more readable code. Async declares a function as asynchronous and await waits for promises to be resolved, pausing execution until the promise is resolved or rejected. This new nomenclature is more modern and avoids the famous “callback hell” (deep nesting of callbacks) of javascript.

Callback hell Example:

function main() {
  getUserData(1, (err, userData) => {
    if (err) {
      console.error(err);
      return;
    }
    getUserOrders(userData.userId, (err, orders) => {
      if (err) {
        console.error(err);
        return;
      }
      getOrderProducts(orders[0].orderId, (err, products) => {
        if (err) {
          console.error(err);
          return;
        }
        processProducts(products, (err, result) => {
          if (err) {
            console.error(err);
            return;
          }
          console.log(result);
        });
      });
    });
  });
}

async/await

async function main() {
  try {
    const userData = await getUserData(1);
    const orders = await getUserOrders(userData.userId);
    const products = await getOrderProducts(orders[0].orderId);
    const result = await processProducts(products);
    console.log(result);
  } catch (error) {
    console.error(error);
  }
}

In summary, Promises are a powerful tool in JavaScript for handling asynchronous operations, providing a clear and structured way to manage the flow of asynchronous code execution. With the introduction of async/await, promise management has become even more intuitive, allowing developers to write asynchronous code that resembles synchronous code in terms of readability and maintainability. Understanding and using promises effectively is essential for developing modern and efficient JavaScript applications.

Thank you for reaching this point. If you need to contact me, here is my email: joaopauloj405@gmail.com

The interesting regex for Identifying Prime Numbers

João Paulo — Fri, 28 Jun 2024 13:33:34 +0000

Prime numbers are numbers greater than 1 that have no positive divisors other than 1 and themselves.

The regex is ^1?$|^(11+?)\1+$

Let's breaking Down...

First you need to know the meaning of these 3 symbols:

^: start of the string
$: end of the string
|: OR operator

So we have two components in the main regex:

1?: The literal “1” and the question mark "?" for match the previous char zero or one times. So if we have zero characters or “1” it will match. Soon we will know the reason for this pattern here.

(11+?)\1+: The second pattern. (11+?) is a group and matches any string that starts with “11” by one or more “1”s. The “+?” makes it non-greedy, meaning it matches as few characters as possible.
\1+ capture the same text again with as many characters as possible.

E.g. So for the number '111111', the pattern '11' is repeated three times. For the number 5 ('11111'), there is no way to split it into repeated sub-patterns.

So the interesting thing is that we found how to evenly divide repeated sub patterns.
In the same number '111111', the pattern '111' is also repeated twice and this is captured by regex.
For prime numbers, the string cannot be divided evenly into repeating sub patterns.

Ah, and first pattern (1?) handles the non-prime cases of 0 and 1.

Thank you for reaching this point. If you need to contact me, here is my email: joaopauloj405@gmail.com

Sapere aude

Understand Dependency Injection in NestJs in a simple way

João Paulo — Fri, 21 Jun 2024 14:01:44 +0000

Dependency injection is when an object provides the dependency of another object. This means that a class relies on methods from another class. It is a well-known design pattern, and Nest is built strongly based on this pattern.

Inversion of control is a concept that aligns with dependency injection. It is basically delegating to the framework the construction of the project's dependencies.

The purpose of injection is to invert control, providing the dependencies of the object instead of allowing it to create them itself. For this, it is necessary to:

Explicitly define the dependencies and where the injection occurs. In Nest, for example, this point is in the constructor. This is where the framework inverts control. constructor(private catsService: CatsService) {}
Have an inversion of control container for dependency injection to occur. In Nest, it is possible to access this container through the module and retrieve the instance of the class we want.

In short, what is done is transferring the task of instantiating the object to someone else (IoC) and directly using that instance as a dependency (Injection).

Advantages of using this pattern:

Ease of testing the application (we can inject mocked services)
Improved code maintenance (reduction of code and decoupling)
Reusability (just inject the dependency to use it)

Disadvantages: There are few, but generally:

Increased complexity in the project
Service construction logic can be in inaccessible places

Thank you for reaching this point. If you need to contact me, here is my email: joaopauloj405@gmail.com

References:
https://www.digitalocean.com/community/tutorials/a-guide-on-dependency-injection-in-nestjs
https://martinfowler.com/articles/injection.html
https://www.youtube.com/watch?v=GX8YC21qQQk&t=1010s