Forem: Tiago Rosa da costa

Holding the load - Avoiding duplicated Webhook requests

Tiago Rosa da costa — Wed, 21 Jan 2026 02:54:45 +0000

What is the motivation?

Imagine the scenario where you have a Webhook integration and the application responsible to notify your application sent the Webhook request more than 1 time.

The application will take the webhook data to create a subscription, so in this scenario will create 2 subscriptions where it will change the user 2 times instead of 1 time, so the user will be angry because of it.

How to resolve it?

Short answer is idempotency key.

The solution applied is converting the data sent on Webhook request to md5 hash and use the md5 hash as id to store on the database. For example:

The webhook request data: { “message”: “test”}

Md5 hash: 42cc32636e077687972862938d538929

So everytime I send the same Webhook request data will generate the same id and I using the column id as PRIMARY KEY, important point about PRIMARY KEY constraint is: each value needs to be unique on column is PRIMARY KEY instruction, save the first Webhook request, but next Webhook request with the same data doesn’t store because already exists.

Summary

When I convert webhook request data to md5 hash to use as the id combined by column id PRIMARY KEY instructions on the database to make sure the column id has only unique values, that way I prevent duplicated Webhook requests.

Link of project: https://github.com/tiago123456789/holding-the-load

Holding the Load: Handling Webhook Traffic Spikes Without Scaling Your cheap VPS

Tiago Rosa da costa — Tue, 13 Jan 2026 11:44:06 +0000

Webhook-based architectures are everywhere.

From payment providers to automation platforms and SaaS integrations, webhooks are often the primary way systems communicate asynchronously. They work well until traffic spikes hit a self-hosted environment.

This post explains Holding the Load, a project I built to solve a very specific but common problem:
how to absorb webhook spikes without vertically scaling a VPS.

The Problem: Webhooks Are Bursty by Nature

If you self-host applications or automation tools, you’ve probably seen this pattern:

A VPS handles normal traffic just fine
Webhooks arrive in short bursts
A spike happens (campaigns, batch events, retries, provider issues)
CPU and memory usage explode
Requests fail or time out

Most webhook providers don’t care about your infrastructure limits. They will:

Retry aggressively
Send large volumes in a short time window
Assume you can handle it

The usual response is to scale the VPS:

More CPU
More memory
Higher monthly cost

But here’s the issue:
That extra capacity is often needed only for minutes or hours, not 24/7.

The Core Idea Behind Holding the Load

Holding the Load introduces a decoupling layer between webhook ingestion and processing.

Instead of letting webhooks hit your VPS directly, you place Holding the Load in front of it.

At a high level:

Webhook Provider
       |
       v
Holding the Load (buffer + control)
       |
       v
Your VPS (consumer)

This separation is the key to stability.

What Is Holding the Load?

Holding the Load is a lightweight application designed to:

Receive high volumes of webhook requests
Store them temporarily
Expose a controlled consumption mechanism for downstream services

Your VPS no longer reacts to traffic spikes.
Instead, it pulls messages at a rate it can safely handle.

How It Works (Technically)

1. Webhook Ingestion

Webhook requests are received by Holding the Load
Requests are acknowledged immediately
Payloads are persisted (FIFO ordering)

This protects webhook providers from timeouts while isolating your backend.

2. Storage as a Buffer

Holding the Load acts as a queue-like buffer:

Incoming webhooks are stored in Durable object(service from Cloudflare) using sqlite storage, so preventing lose the webhook data.
Order is preserved
No processing happens at ingestion time

This is critical: ingestion and processing are completely decoupled.

3. Controlled Consumption by Your VPS

Your application:

Pulls messages from Holding the Load
Defines:
- Batch size
- Pull interval

Example:

Fetch 10 messages every 5 seconds
Or 50 messages every minute
Or any strategy that fits your VPS capacity

The first webhook received is always the first consumed (FIFO).

Why This Architecture Matters

This approach solves multiple problems at once:

✅ Traffic Spike Absorption

Webhook spikes are handled upstream without affecting your VPS.

✅ Predictable Resource Usage

Your VPS workload becomes stable and predictable.

✅ No Overprovisioning

You don’t need to pay for peak capacity all month long.

✅ Failure Isolation

Even if your VPS goes down temporarily, webhooks are not lost.

Serverless Cost Model

Holding the Load follows a serverless-style philosophy:

Resources scale based on demand
You pay only for actual usage
Idle time costs almost nothing

This is particularly useful when:

Spikes are rare but intense
Traffic patterns are unpredictable
You want cost efficiency without sacrificing reliability

Typical Use Cases

Holding the Load works especially well for:

Automation platforms like N8N
Self-hosted workflow engines
Api
Ai agents where react based webhook event

Why I Built It

I built Holding the Load after noticing a recurring pattern in self-hosted systems:

We scale infrastructure to handle rare peaks, not real workloads.

Holding the Load flips that logic:

Keep the VPS small and cheap
Scale only the ingestion layer
Let processing happen at a controlled pace

Final Thoughts

Holding the Load is not a replacement for queues, workers, or job schedulers.

It’s a protective layer.

A buffer that:

Shields your VPS
Controls load
Reduces cost
Improves reliability

If you rely on webhooks and self-host your infrastructure, this tool can simplify your scaling strategy.

Here’s the updated ending with the project link added cleanly and naturally for a tech blog:

Final Thoughts

Holding the Load is not a replacement for queues, workers, or job schedulers.

It’s a protective layer.

A buffer that:

Shields your VPS
Controls load
Reduces cost
Improves reliability

If you rely on webhooks and self-host your infrastructure, this pattern can dramatically simplify your scaling strategy.

Project Repository

You can find the full source code, documentation, and examples here:

👉 https://github.com/tiago123456789/holding-the-load

Feedback, issues, and contributions are welcome.

Need Help or Want to Talk?

If you’re facing webhook scaling issues, evaluating this architecture, or need help adapting Holding the Load to your own setup, feel free to reach out.

I’m happy to:

Discuss real-world use cases
Help with architecture decisions
Answer questions about the project
Support integrations or custom scenarios

📧 Email: tiagorosadacost@gmail.com

ai #aiagents #automation #n8n #aichatbot #chatbot

Chatbot para ajudar a melhorar a conversão em inglês(gratuito)

Tiago Rosa da costa — Sat, 09 Aug 2025 16:19:49 +0000

Se você está ainda sente que o inglês te trava um pouco — seja pra entender documentações, participar de calls ou aplicar pra vagas fora ou expandir possibilidades de negócios — conheça esse projeto que lancei: EnglishLanguageTutorBotFree

O que é?

É um chatbot no Telegram que usa inteligência artificial pra simular uma conversa semelhante a uma conversa com o professor de inglês. Você envia áudio ou texto, conversa em inglês, e no final recebe feedback automático com pontos pra melhorar. OBS: essa ferramenta tem como foco ser um complemento no seu estudo de inglês.

Sem cadastro, apenas instalar o Telegram e acessar https://t.me/EnglishLanguageTutorFreeBot

Funcionalidades já existem:

Conversas em inglês por texto ou áudio sobre qualquer assunto de interesse — como se fosse com um professor real.
Transcrição automática dos seus áudios.
Respostas também em português, para facilitar quando você esquecer uma palavra ou quiser perguntar algo como:"How can I say ‘dinheiro’ in English?"
Você pode também pedir para elaborar um aula ou simular um entrevista de emprego. Por exemplo: você pode enviar um áudio ou mensagem falando algo como: 'I want to study more about verb to be, so elabore a quiz about verb to be using the topic football' ou "Let's simulate a interview for Developer or Project manager, so elabore 10 most common questions and play as HR recruiter while I candidate"

Funcionalidades que serão adicionadas em breve:

Lembrar você de estudar todos os dias
Newsletter com links de esportes/lifestyle/notícias pra ler em inglês
Salvar palavras novas pra estudar depois
Gerar Excel com essas palavras pra importar no Anki
Criar tarefas personalizadas com base no feedback da aula
Gerar quizzes e desafios com base nos seus interesses (ex: quiz sobre Brasileirão em inglês pra treinar o verbo to be)

Why Should You Specify a Language Version in Your Dockerfile?

Tiago Rosa da costa — Sun, 05 Jan 2025 14:42:19 +0000

Let me share a lesson learned the hard way. Once, while working for a company, our application was running flawlessly. However, after deploying a new feature update, the application started displaying strange characters in place of properly accented words.

The development team spent countless hours investigating potential causes of the problem. After much effort, I noticed something odd in the Dockerfile: we were using Node.js without specifying a version. By default, Docker pulls the latest version if no specific version is defined.

In our case, Docker had downloaded Node.js version 22.7.0, which happened to have a critical bug affecting UTF-8 encoding. This caused words with accents to render as garbled characters. (For reference, see Node.js issue #54543.)

The Fix

The solution was simple yet crucial: we specified a stable Node.js version (in this case, v20) in the Dockerfile and redeployed the application. Immediately, everything returned to normal.

Best Practices for Your Dockerfile

Always specify the exact version of your base image to avoid such issues. Here's an example:

In the first line, we explicitly set the base image to Node.js v20, ensuring stability and avoiding unexpected behavior caused by breaking changes in future versions.

Bonus Tip: Optimize Your Docker Image

When creating a Docker image, consider using the Alpine version of your base image (e.g., node:20-alpine). Why?

Lightweight: Alpine images are significantly smaller, saving storage space. This is especially important for private repositories like AWS ECR, where costs are tied to storage usage.
Faster Builds: Smaller base images download and build faster, reducing deployment time.

By specifying the base image version and using optimized images, you can avoid unexpected bugs and improve your build process efficiency. Don’t let a simple oversight like omitting the language version derail your application’s stability!

React in MVC application

Tiago Rosa da costa — Sun, 03 Apr 2022 01:47:33 +0000

What is it? I believe you are questioning and this question is because it is uncommon to use react.js this way, in majority the application all frontend is built using react.js and consume api REST.

So, let's go, react.js in MVC application, before getting out utilizing some technology needs to understand the finality and situation more suitable to apply.

Imagine the following situation: the company has MVC application and this application has more than 2 years life, many things already made and you have many feature for build, using jquery will easily the build the feature until react.js is came, where is a lot easily build feature has many interactions. So understand the benefits the react.js you present to your company, all stay interesting, but the company has many things that are necessary to change to react.js. Because there are many things to change, the company denies adoption of react.js.

Many times we programmers stay get excited the technologies and we want apply this technologies all places, but when we working in some company is necessary analyses some points with: community of technology, adoptions the technology per persons, impact the adoption this technology in application, finance impact the adoption this technology and benefits the adoption this technology to the company.

I believe this approach to use of react.js in MVC applications is interesting, because it prevents many changes in application to apply this technology. This way the programmer stays happy per work using new technology and company too, because the productivity rises.

Ok, I speak a lot but show nothing, so, let’s go. Image below is the project structure that I created to explain this approach.

Details about folder structure:

Directory componentsReact is where react.js code. OBS: directory name is different, this does not affect anything.
Directory src where find application web code node.js.
Directory public where find js and css files.
Directory views where find applications page. Now we go into the deep react.js code part. The image below show settings of webpack:

Summary the webpack.config.js file working the following way:

The entry is entry file used per webpack to generate bundle.js
The output is section where settings where webpack to generate bundle.js and put within public directory to use more later in views directory.
When generate bundle.js code is set library variable and attributed bundle.js code to this variable
The module is a section where has rules and loaders necessary to webpack handler javascript new versions and css code in react.js components.
In ./src/index.js file inside componentsReact directory necessary stay this way:

Let’s go, explain image above:

You saw there exist two functions renderLoging and renderLogout both have parameters, the container receive element html id where I inject react component into, already seconds params receive one object where is passed with props to react component.
In the end file export functions that were added in bundle.js file.
You must stay thinking about how to render the react component on one page. The image below shows how to make this:

Summary what is happening here:

Import bundle.js file in page.
After create script block where call variable defined in webpack.config.js that has two function exported. In line 45 is called renderLogin function passing information.

The result is:

One thing important is http requests made per react.js, no necessary implement jwt authentication with jwt token, because when javascript makes a request the same domain automatically sends cookies in request this way you can identify the user authenticated per session.

Case you have interesting about the project used, link of repository here: link of github

Summary:

In this post I want to pass one approach different from how to use react.js. Another thing I believe has passed is technology is tools and you need to analyze the impact that it can cause in application.

In this example I use application node.js, but you can use this approach to applications laravel, spring, ruby on rails, etc

Scalability web application

Tiago Rosa da costa — Sun, 03 Apr 2022 01:34:04 +0000

Scalability is the capacity the application handles growth access number, users and handler data volume without affecting user experience.

If you search about scalability in web applications, you will probably find someone with: vertical scalability and horizontal scalability, you can think what is it first moment? Easy my friend, keep going, I will explain to you what I know about this theme.

Vertical scalability

This scalability type where you increment hardware resources in the machine where running your application. Increase hardware resources with: cpu, ram and disk to handle growth access number, users and data volume.

The image below describe the vertical scalability well:

Increasing the hardware resources of a machine is relatively easy when you are using IAAS(infrastructure as a service) with AWS, Azure and DigitalOcean, because you access the specified service and change machine settings in a few minutes.

But has some points you need warning:

Exist the hardware resource limit you can increment one machine
If there is some problem on the machine or in the data center where the machine is put. Your application can't be accessed.
Sometimes you have a big machine, but no need for all hardware resources, you need only moments of peak access.

Horizontal scalability

This scalability type where you increment machine numbers where each machine has an application running to handler growth access number, users, data volume.

Has one component very important that is load balancer responsable distribuite request between machines with application web running.

The image below describe horizontal scalability well:

In this case only creating one machine with an application is when the ready load balancer sends a request to this new machine.

One thing very interesting this scalability type if you settings yours machines in diferentes availability zones(AWS concept) you have fail torelancy, because if one machine or data center
has some problems if your application keeps running because there exist other machines in other availability zones(AWS concept).

If some IAAS with AWS has autoscaling, that is one service that allows increment or decrement number machines automatically based on some settings. For example: when the machine group reaches 80% cpu usage, create a new machine, and when it reaches 30% cpu usage remove one machine.

But has points you have warning:

If your application uses sessions authentication is necessary, enable sticky sessions on load balancer.
If your application has upload of files is necessary, use services with S3 to storage your files, because if you store in disk the machine sometimes can have problems. For example: you have 2 machines, you make request load balancer send you to machine 1, you store file machine 1 disk, moments later you can need get file stored but you request load balancer send to machine 2 where don’t have this file in disk and to resolve this problem you some service with S3.

Vertical scalability or horizontal scalability? Which is better?

Your application has a fixed access number customer and the application don’t will grow up, so vertical scalability. If you don’t have a certain access number and can increment access number, users and data volume, so horizontal scalability.

Serverless architecture

Tiago Rosa da costa — Mon, 07 Feb 2022 16:10:37 +0000

What’s serverless architecture?

This serverless architecture has the following characteristics: pay to resources used and cloud provider is responsible per running and scale out and scale in based needs.

Another thing word serverless means no server, no true because you need one server to run code, in this case i believe this name is to call focus and because too no need know server where is running your code.

Many times when people talk about serverless architecture the lambda function is referenced, but any anothers services can be considered serverless for example: firebase, s3, faunadb and more.

Evolution until serverless architecture

In house

In house many time before people running servers in house where all responsibility the company, you need take care internet link, electricity, maintain OS updated and another responsibilities. But one big disadvantage is when you need to scale your application you need to increment hardware or purchase another server machine takes a lot of time. This is one problem because you need to scale your application to spike at 12pm for example.

IAAS

IAAS is knowledge with infrastructure as a service where many cloud providers with: aws, gcp and azure offer services. Using IAAS you pay per resource that you need for example: if you need running one web application you use an ec2 instance.

IAAS allows the choice of resource hardware or modify resource hardware quickly and easily. In Many cases you need to scale your application when receive spike access at 12pm exist ways allow create copies of the web application in other machines quickly.

PAAS

Paas is knowledge of the platform as a service, where Heroku is a good example. The paas is one evolution of the IAAS. In this case you specific technology and send code to paas, paas is responsible for setting all things to your application running. Removing your responsibility settings all things to your application running.

Serverless

Services model serverless pay what you use, different another solution above where you pay to the service running same no any interaction with service. The model you pay to use and scale up and down is the responsibility of the cloud provider. For example: firebase you pay what you use and if volume grows the firebase increases resource it or S3 is another example you pay to the file stored and download the file no need to worry about disk full because this is the responsibility of the s3 service.

What's a lambda function or FAAS?

The lambda function is one function of any programming language where one specific task is executed.

I think it is necessary to talk about lambda function or FAAS(function as a service) because many times when people talk about serverless architecture already associated with the lambda function, but when analyzing aspects of the serverless architecture you pay what you use and cloud providers manage your application for you. When I think about this, Firebase and S3 are serverless.
When and why use the lambda function?
Imagine that you have a service responsable send email to anyone, to prevent create all applications to only to send email and pay to run this application one ec2 monthly same application no receive any interaction some moments, already another moment you have huge volume the emails to send and you need scale this application to delivery all emails suitable time.

In this situation the lambda function is a great solution because you need to create only one function to send email, pay only what you use and the cloud provider scales up and down automatically the lambda function numbers when increasing the volume of email to send.

Too when you need execute task spend little time less than or equal 15 minutes, tasks eventually have spike execution and other times no have interactions and you don’t care responsibility the scale up and down your application you want give this responsibility to cloud provider and you focus logic the application.

When not using the lambda function?

You need to generate a report that takes a lot of time to generate. In this case no suitable use of the lambda function because the lambda function limit lifetime is 15 minutes in aws.

The websocket application is another example because after time the lambda function process is finished, so, websocket clients lose connection.

The api that uses database no scale resources automatically. The lambda function scales the application layer, but your database does not scale resources automatically. It's one problem, because each request creates one the lambda function execution each the lambda function creates a new connection to your database, if your database is not scale you can down your database. In this case is interesting use one database with scale based on demand like: dynamodb, faunadb or aurora serverless

What is the lambda function cold start and why take care about it?

Firstly let me explain what is the lambda function cold start, the cold start is time needed to the cloud provider in this case AWS get the lambda function code and put a docker container with code to run and after execute what you need. This occurs on the first time you interact with your lambda, after interactions no have cold start until the cloud provider kills the lambda function because no have interactions, in case this occurs the next interaction the lambda function will have the cold start again.

Now you are thinking what is the problem the cold start because only increase time little bit, if is not problem for you, ok, but exist situation where you have specific response time for example 5 seconds to response one request, in this situation to reduce the cold start execute following things:

Use only dependencies necessaries because more and more dependencies more size will be the lambda function code more time spend put it to running in docker container
Increment resources the lambda function only when necessary. Example: memory
Using interpreted language like python, node.js and another because check types in runtime.
Warm up the lambda function this way time to time interacte the lambda function to maintain warmed up. In this link https://www.serverless.com/blog/keep-your-lambdas-warm/ you have one example about this.

The lambda function disadvantages?

Vendor lock.
Limit timeout 15 minutes.
Limit memory 10GB.
Can’t store files in disk because you will lose the file.
High volume connections because create one execution of the lambda function that creates a new connection on the database.
More complex implements tests in application.
You will have problems when using some dependency OS, because the lambda function running in docker, docker container is light. So install only packages necessaries.
Development environment needs components from the cloud provider.
Deploy the code manually or you need to use cloudformation or terraform.

The lambda function advantages?

A lambda function is one function isolated from another code in the application. When you change code the lambda function has no impact on another lambda function that occurs in a monolith application.
The cloud provider is responsible scale up and down After deploying the application your team no need managed it is the responsibility of the cloud provider.
Low cost because you pay what you use

How to deal with some disadvantages of the lambda function?

Can’t store file in disk: because after time execution the lambda function finishes and you can lose file storage in disk. To resolve this problem using storage solutions like S3.

Development environment need components of the cloud provider: exist a project that simulates the cloud provider services locally, this project named localstack. https://github.com/localstack/localstack

Deploy the code manually: you can resolve this problem using a serverless framework. This tool simplifies your deploy because this tool reads serverless.yml file where all needed settings are to deploy the lambda function. Link of serverless framework: https://www.serverless.com/

High volume connections because create one execution of the lambda function that creates a new connection on the database: in this situation you need some database to scale your resource on demand like Dynamodb.

More complex implements tests in application: the lambda function is an event trigger, so when one event triggers the lambda function executes code with the event data. In this case you can implement unit tests mocking event the lambda faction will receive to prevent use of the cloud services.

Aspects you need take care before build all thing using lambda

Take care how to use the lambda function because the cost is the sum of the following things: how many times executed the lambda function, time each execute the lambda function code and resource allocated to execute this lambda function.

Another thing necessary take care is when you use the lambda function you scale application layer, but some moment this capacity the application layer scale can be cause the problem in database layer because create many connection on database or you communicate third service and third service have limit request per time or third service no prepare receive huge volume request, case no prepared you can down third service.

These are something you stay warned about when you think about using the lambda function(serverless architecture).

Summary

I believe this post I explained a little bit of what I know about the serverless architecture, what it means, and explains that services like Firebase or S3 are serverless, not only the lambda function(FAAS).

Talked more focus about the lambda function, when and where to use it, when not to use it and explain the advantages and disadvantages and keep in mind it is not a silver bullet, exist sceneries correct to use it, sceneries wrong to apply the lambda function.

Quick tip - S3 sync

Tiago Rosa da costa — Sun, 18 Jul 2021 01:56:06 +0000

What’s s3 sync? The s3 sync is one command of the aws-cli which allows synchronize files with a bucket. The complete command is: aws s3 sync ./ s3://bucket_name --delete

Explain in details this command:

The part aws s3 sync indicate to aws-cli that you need synchronize something with s3
The part ./ get all files in current directory to send to s3
The part s3://bucket_name change bucket_name per name the bucket that exists in your aws account. This bucket receives files synchronized.
The part --delete is optional. When mentioned this parameter indicates that after synchronization occurs delete files in the local machine.

I believe that following situations s3 sync can help you:

Your application running in machine and write log files in disk, but disk is full and you can’t delete log files of your application.
You configured your database and made backups of the database because you didn't use management database service You deploy the react application in s3.
You execute build and upload files in s3 where s3 hosted frontend application.

Let’s talk about the situations above and explain how s3 sync can help you.

First situation: your application running in machine and write log files in disk, but disk is full and you can’t delete log files of your application

In this scenery your application writes log files in disk, ok. But it is necessary to remove these files because the machine disk is full and can’t delete log files. One solution to resolve this problem:

aws s3 sync ./ s3://bucket_name --delete

Explain command above:

Firstly install aws-cli and configure credentials.
After execute command above where get files in ./ , send to bucket_name and delete local files.
To automate is necessary only create cron task in machine to execute command above based your needs.

Second situation: you configured your database and made backups of the database because you didn't use management database service

In this scenery you need to generate backup daily from the database and store one place to another moment access, case needs. The solution to resolve this problem:

mysqldump -u username -p database_name > ./backup_$(date +"%m-%d-%Y").sql
aws s3 sync ./ s3://bucket_name --delete

Explain commands above:

Firstly install aws-cli and configure credentials.
After use first command above in this example to execute backup mysql database.
After use second command to get files in ./ , send to bucket_name and delete local files to prevent machine disk full.
To automate is necessary only to create a cron task in the machine to execute the commands above.

Third situation: you deploy the react application in s3. You execute build and upload files in s3 where s3 hosted frontend application

You deploy a manually react application in s3 where s3 hosts a frontend application. The solution to resolve this problem deploy manually:

npm run build
cd ./build
aws s3 sync ./ s3://bucket_name

Explain commands above:

Firstly install aws-cli and configure credentials
After use the first command to execute build react application that generates build directory when contains files to deploy.
After use second command to access build directory
After use command to get files in ./ and send to bucket_name
To automate it for you, you can use github actions to execute pipeline. When sending code to the branch master in the github repository execute the github action pipeline with command above. To github actions pipeline use this https://github.com/aws-actions/configure-aws-credentials where setting aws-cli and configured credentials in your aws-cli and after execute commands above

Webhook

Tiago Rosa da costa — Sun, 25 Apr 2021 01:00:43 +0000

What is it?

Is one request http using the verb post that is triggered when one event in the application to notify another application.

When do you use it?

One example is api payment where use webhook to notify a third application about status payment. Imagine an api payment implemented one form communication for each client, it’s impossible to implement and maintain it.

So, when your application needs to send data for a third application when an event occurs and needs one simple form to make this, webhook is one good solution for it.

How to work?

I go use an example of the payment api. When a client create one transaction him informed webhook url and payment api when change status the transaction send data for webhook url informated.

Image below illustrate what i spoke:

How to make webhook safe?

The webhook url is public, so anyone can access this url. To make webhook url more safe enable https in webhook url and add secret in webhook url for when application trigger webhook url the third application will check if secret is valid, if yes, process request, case no, reject request. Example webhook url with secret: https://domain_url/rota?secret=value

What situations apply webhook?

The first scenery using webhook url to create CI/CD pipeline, explain flow:
- Add webhook url the jenkins for trigger when push code in repository
- When webhook url is triggered execute one job
- The job run tests the project and after trigger another job
- The second job to deploy changes in staging or production server.
The second scenery using webhook url is automate deploy, explain flow:
- Add webhook url the for trigger when push code in repository
- When the webhook url is triggered execute one code that changes the project in staging or production server.
The third scenery using webhook url is one course platform, I will explain flow:
- People buy course using credit card
- Send datas payment api
- When payment is approved it triggers a webhook url on the course platform to enable courses for people.

The example application

In file index.js is code example:

Explaining image above:

In this file I’m use express for create application and define route /webhook
In line 9 until 20 I’m check if request has a token valid, case yes, process data, case no, no process.

In the github you set a webhook url. Image example:

Explaining image above:

In Payload URL you set url and add token. Example: webhook_url/route?token=value_token
In Content type set application/json to application webhook receive data in format json.
In section “Which events would you like to trigger this webhook?” I set an event trigger webhook, this case when I make push code for a repository.

When you have one application need handler many webhooks is necessary think about one strategy for scale up and prevent that affect performance the principal application, because when application make request third application this third application can low performance and it is affect your application.

The image below is a solution for this problem is:

Explaining image in above:

Actor create transaction api payment
When occur one event that is necessary trigger webhook, send data for message queue
Another side has a job to get data and make request to notify third application.

So, my friends, here I finish one more article, case you have one question about webhook, comment below I try reply with one solution.

Link the project repository in Github: https://github.com/tiago123456789/webhook-article-project

Quick tips - Docker, Dockerfile and Docker compose

Tiago Rosa da costa — Sun, 11 Apr 2021 23:23:17 +0000

Docker

What is?

Docker is one platform that allows running containers. The containers are processes where in this process have application and libraries necessary to run application.

Why use this?

Someone one moment said “to work in my machine” this problem occurs because when configuring applications in staging or production environments can have different versions of OS, libs and softwares used in the application. Docker allows specific version OS, libs and softwares prevent this problem mentioned above.

Virtualization or docker?

The virtualization will be installed on another OS over the machine, another side the docker is different from the virtualization where the docker uses the kernel of linux to run the processes docker called containers. The containers only necessary to run applications make containers light compared to virtualization.

How to install?

To install docker use this link: https://docs.docker.com/get-docker/ in this link have instructions to install in windows, mac and linux. After install, back the article. Let’s begin.

Check docker installed:
docker --version

Run docker container com nginx:
docker run nginx

Explain command above:
The command docker run é usando para criar um container docker
The name nginx é uma image docker. The image docker is one file where have instructions to configure nginx and your libraries.

Run docker container com nginx:
docker run --name=value_name nginx

The parameter --name=value_name is used for the set name the docker container.

Run docker container nginx:
docker run --name=value_name -p 8080:80 nginx

The parameter -p 8080:80 sets the docker containers allowing that when access port 8080 the host machine redirects for port 80 in docker container.

Run docker container background:
docker run -d -p 8080:80 nginx

The parameter -d in the command above is one indicator to run the container in background. Allowing the same terminal execute another action.

Stop container docker is running:
docker stop id_name_container

List containers docker running in moment:
docker ps

List containers stopped:
docker ps -a

Remove container docker stopped:
docker rm id_or_name_container

Remove container docker is running:
docker rm -f id_or_name_container

Access and interact with container docker is running:
docker exec -it id_or_name_container command_execute_container

Explain command above:
The parameter -it will indicate you want interact with container docker
In place command_execute_container you can use the /bin/bash open terminal to interact with the container docker.

Using environment variable in docker container:
docker run -e KEY_NAME=VALUE image

The parameter -e after KEY_NAME=VALUE the flag -e is used when a set environment variable is needed for the docker container.

Using volumes in docker container:
docker run -d -v path_local_machine:path_in_container -e KEY_NAME=VALUE image

What is volume?

The volume is one functionality that allows persistent data in applications running in docker. Example: you execute command for running your application and the application write logs in file, case occur docker container died the log file is deleted, but you need maintain this log file same docker container died in this case volume is the solution, because when you execute a command to create docker container your application you set the volume this allow that log file write in path mapped in container is write in local machine, case docker container died file keep persisted in local machine. When docker container restarts, it gets a log file in the local machine and synchronizes the docker container.

List images docker in machine:
docker images

Download image docker the docker hub:
docker pull name_image

The docker hub is repository images docker is equal to github, but the docker hub focus store image docker.

Remove image docker the local machine:
docker image rm image_name

Dockerfile

When you work in one application you have specific needs and it’s make needs you create your own Dockerfile. If you don't know what is docker until moment, calm down, I'll explain better about it.

Image Dockerfile with cake recipe where I have necessary instructions to make cake, in this case cake is container docker.

I’m imagining the now concept about dockerfile is ok. Let’s go example:

FROM node:15.13.0-alpine3.10

ENV PORT 4000

WORKDIR app/

COPY . ./

RUN npm i 

EXPOSE $PORT

CMD node server.js

Example step by step Dockerfile:

Dockerfile is text file
The parameter FROM is the start point in Dockerfile. This case I’m using node image with base. Case we want create one image absolute zero, replace node use a image scratch The parameter ENV is a one form set environment variable. In this case I’m creating one environment PORT with default value 4000.
The parameter WORKDIR is used to set a directory to work. In this case set directory app/
The parameter COPY is used as a copy file in the local machine for docker containers. In this case I copy all files local machine for docker container in directory app/
The parameter RUN is used to execute commands with if typing command command line in terminal. In this case, I executed npm i to install dependencies on the project.
The parameter EXPOSE is used more with documentation, because no expose this port. In this case the parameter is to indicate that necessary exposure port 3000.
The parameter CMD is used to execute when docker container created

Create one image based Dockerfile:
docker build -f ./Dockerfile -t name_image:version .

Explain command above:
The parameter -f is used to set a path to your Dockerfile.
The parameter -t is used to set name, image and version.
The ‘.’ on the final command used for the specified current directory.

Send image for docker hub:
docker login --username=username_docker_hub
docker tag id_image_docker username/project_name:version
docker push username/project_name:version

Explain commands above:
The first command is used in docker hub(image repository).
The second command is used to create one tag for one image.
The third command is used to send images for docker hub.

Link the project in github: https://github.com/tiago123456789/project-for-article-docker

Docker compose

What is it?

Docker composer is one tool that allows management of docker containers the simple way and prevents execute a lot of tasks in hand.

How to work?

Before keeping reading this article using this link https://docs.docker.com/compose/install/ para install docker compose, after installed docker compose is needed create file called docker-composer.yaml where will be add instructions to run containers. The file docker-compose.yaml is yaml one file text what to work using indentation.

Below is one example of docker-compose.yaml file:

version: "3"

services:

    app: 
        build: ./
        container_name: app_example
        environment:
            - PORT=3000
        ports:
            - 3000:3000

Example code above:

Line 1: is set version
Line 3: from this line below is the service section where the docker containers that docker compose will be management.
Line 5: set name the service.
Line 6: using word build and set ./ this is one indicator mentioned docker compose to using Dockerfile with image this service.
Line 7: set name the container
Line 8 and 9: set environment variable used in application running into container.
Line 10 and 11: set port your local machine and port application running into the container. When made a request in the local machine will redirect to a container running in this port.

Now, after define docker-compose.yaml is need run containers.

Running docker containers using docker-compose:
docker-compose up

Running docker containers in background using docker-compose:
docker-compose up -d

Remove docker containers created using docker-compose:
docker-compose down

I think that to the initial article about docker, it’s ok. Case you want to learn more about docker, comment below.

Improving performance the web application

Tiago Rosa da costa — Wed, 03 Mar 2021 15:01:48 +0000

Improving performance the web application

Pagination: looks simple, but exists many applications no applying pagination in listing the data. Returning all registers generate processement unnecessary in many situations, not is necessary to return all data in one only time.

Orm: are tools that abstract communication with databases to the form object oriented, but there are moments where it is necessary to execute complex queries with orm is slow. This moment use native sql to complex queries is the correct for gain performance.

Search only the necessary: one another thing a lot simple, but when one application grows up it’s simple detail makes different. Imagem the scenery: you make one request to one endpoint in api and return many data unnecessary for context present generated process unnecessary. This moment can use one querystring to specifically necessary to return in response. Example: /peoples/?fields=usename,email.

Static files: are files that no matter how many times you request the files, the content continues the same. The css, js and image files are good examples. When we speak about static files there are some solutions to improving performance:

Minify css and js files this form the file size reduce and it’s allow download more quick.
Use CDN, what is it? CDN(content delivery network) where static content is cached and the content is searched in one machine geography more near the person that requested it.

Zip the response: when we work with applications using http protocol to request something, make requests to the server and the server processes data and returns a response. The response can be zipped making response reduce size and allowing download more quickly.

Asynchronous process: are processes that can execute in a moment later, different the processes synchronous that need execute in exactly the moment. Example: imagem that after registering a new user must send the email welcome. This email no need be sent in exactly the moment register user, email can be sent one time later.

Cache: is one solution very interessent to improving the performance of the applications due the data stored in memory. Example: one user makes a request to your application, you search the data in the database and store it in cache and after returning a response to the user. In the next request no necessary search data in the database only gets data in cache. It makes response return more quick, because getting data in memory is more quick than getting data in the database.

Essential commands pm2

Tiago Rosa da costa — Fri, 26 Feb 2021 14:49:36 +0000

Essential commands pm2

I think that before we need to understand what is pm2? Pm2 is one tool that allows management of the process, case the process stops running the pm2 make restart the process, another thing interessent is that you can look at logs of the process to understand error that occurs.

Command to install pm2:

 npm i -g pm2

Command above install the pm2 globally

Command listing the processes running in pm2:

 pm2 list

Command to execute one process with the pm2:

pm2 start path_script_js

Command to execute one process and specific name to the process with the pm2:

pm2 start path_script_js --name=name_process

Command to execute one process in cluster mode with pm2:

pm2 start path_script_js -i max

The command above init script and the parameter -i max will put this process the same as how many cpu core of the machine is running the application. It's to prevent that you have to implement it manually using a cluster module of node.js for creating child processes starting the parent process. It’s one strategy used to better performance in node.js applications.

Command to stop all the processes in the pm2:

  pm2 stop all

Command to stop one process in the pm2:

pm2 stop id_process

Command to delete all the processes in the pm2:

pm2 delete all

Command to delete one process in the pm2:

pm2 delete id_process

Command to see logs one process in the pm2:

 pm2 logs id_process

Command to restart one process in the pm2:

  pm2 restart id_process

Command to restart all the processes in the pm2:

 pm2 restart all

Command to save the processes in the pm2:

pm2 save

The command above is helpful, because will create one backup file with the processes that are running in pm2, case have that make reboot the machine the pm2 will use the backup file to recreate the processes.