Forem: Lucas Aguiar

Understanding MongoDB $lookup performance

Lucas Aguiar — Fri, 05 Sep 2025 00:30:01 +0000

After running into some issues with $lookup in our banking application, I decided to dive deeper into how it actually works and why it sometimes ends up being so slow.

The problem

It was just a normal day. I was working on a new feature when my tech lead mentioned me on Slack with something like:

“Lucas, your query is too slow”
(and then pasted the Explain Plan)

Right after that, he casually added: “This query took 33 minutes to finish.”

In my head, I was like: “Wow… how did this slip through? The code has been there for three months already!”
That’s when I decided to investigate the problem more deeply.

Here’s the query he sent me:

[
  {
    "$match": "simple start filtering"
  },
  {
    "$lookup": {
      "from": "Company",
      "let": {
        "company": "$company"
      },
      "pipeline": [
        {
          "$match": {
            "removedAt": null,
            "$expr": {
              "$and": [
                {
                  "$eq": [
                    "$_id",
                    "$$company"
                  ]
                }
              ]
            }
          }
        }
      ],
      "as": "company"
    }
  },
  {
    "$lookup": {
      "from": "Customer",
      "let": {
        "customer": "$customer"
      },
      "pipeline": [
        {
          "$match": {
            "removedAt": null,
            "$expr": {
              "$and": [
                {
                  "$eq": [
                    "$_id",
                    "$$customer"
                  ]
                }
              ]
            }
          }
        }
      ],
      "as": "customer"
    }
  },
  {
    "$match": "more filtering"
  },
]

Right after reading the query, my first thought was: “Why are we using $expr and pipelines inside the $lookup?”

So the very first thing I did was paste this aggregation pipeline into MongoDB Compass to run an explain and try to visualize why the query was so slow.
And to my surprise… I just got an infinite loading screen 😅

I promise, I waited a long time hoping to see some output. But in the end, all I got was a timeout error.

At that point, I decided to break things down step by step, isolating each stage to figure out where the bottleneck was hiding.

Testing

First, I ran an explain on just the $match. Everything worked perfectly: no problems at all. The query took only 10ms and used a simple company index.

[
  {
    "$match": {
      "removedAt": null,
      "company": ObjectId("some_ID"),
      "type": "TYPE_FILTERING"
    }
  }
]

So far, so good.

Next, I added the first $lookup and here’s where things started to smell funny. The execution time jumped from 10ms to 314ms.

Before moving on, I decided to test both $lookups together, just to confirm where the problem was. And… boom! 6 minutes to return the result. So clearly, the issue had to be in the $lookups.

Understanding the lookup

Everything here are based on the official documentation of lookup on mongodb.

So, to start, $lookup has this syntax and these fields

{
   $lookup:
     {
       from: <collection to join>,
       localField: <field from the input documents>,
       foreignField: <field from the documents of the "from" collection>,
       let: { <var_1>: <expression>, …, <var_n>: <expression> },
       pipeline: [ <pipeline to run> ],
       as: <output array field>
     }
}

At the basic, the easy way to use an lookup is with localField and foreignField, but on our query we are using let and pipeline to find the results.
But why? At the application we use soft delete to prevent deleting important data by an error, so everything keeps stored but with removedAt flag, but we really needs to filter with pipeline?

Understanding the problem of `$expr`

According to the official $expr documentation:

When $expr appears in a $match stage that is part of a $lookup subpipeline, $expr can refer to let variables defined by the $lookup stage

The problem here is that when MongoDB applies the filter, it has to evaluate the condition document by document inside the collection. As the collection grows, this operation gets slower and slower, since it can’t rely on indexes the same way a direct match does.

In other words: $expr inside a $lookup pipeline forces MongoDB to do more work, which quickly becomes painful at scale.

The better approach is to use localField and foreignField whenever possible. That way, MongoDB can take advantage of indexes efficiently. For example:

  {
    "$lookup": {
      "from": "Company",
      "localField": "company",
      "foreignField": "_id",
      "pipeline": [
        {
          "$match": {
            "removedAt": null,
          }
        }
      ],
      "as": "company"
    }
  },

And here we go! 🎉
Now the response time is totally acceptable:

Improving the pipeline even more

After some experiments, I discovered that whenever we use a pipeline inside a $lookup, the query slows down a lot sometimes 5x to 10x slower.

So, if your application doesn’t really need extra validations, I strongly recommend avoiding the pipeline. Instead, stick to localField/foreignField. This is much simpler and far more efficient when you’re just joining on _ids.

In my case, by removing the unnecessary pipeline, I improved the query from 259ms down to 36ms. Here’s the simplified version:

[
  {
    "$match": "simple filtering"
  },
  {
    "$lookup": {
      "from": "Company",
      "localField": "company",
      "foreignField": "_id",
      "as": "company"
    }
  },
]

Conclusion

After all the tweaks, the final query ended up looking like this:

[
  {
    "$match": "simple filtering"
  },
  {
    "$lookup": {
      "from": "Company",
      "localField": "company",
      "foreignField": "_id",
      "as": "company",
    }
  },
  {
    "$lookup": {
      "from": "Customer",
      "localField": "customer",
      "foreignField": "_id",
      "as": "customer",
    }
  },
  {
    "$match": "more filtering"
  },

And that’s it a (not so short) story, but with some practical tips on how to improve $lookup performance:

✅ Prefer localField / foreignField whenever possible
⚠️ $expr can slow down queries a lot
🛠️ For more complex lookups, combine pipeline with localField/foreignField

Understanding IMAP and SMTP Mail Protocols

Lucas Aguiar — Sat, 26 Jul 2025 01:48:38 +0000

During a recent implementation of a customer service platform, I encountered several configuration challenges related to email communication protocols. This experience prompted me to conduct thorough research into SMTP and IMAP protocols, their interactions, and best implementation practices. The knowledge gained proved invaluable for resolving our technical issues, and I'm sharing these insights to benefit others facing similar challenges in their systems architecture.

📮 Introduction

In the digital world, sending and receiving emails are fundamental communication processes. During my recent experience, I realized that understanding how these protocols work behind the scenes can make all the difference, especially when configuring or troubleshooting customer service systems.

🌐 Overview of Email Protocols

Emails travel between computers using standardized protocols. The most common ones are:

SMTP (Simple Mail Transfer Protocol): Used for sending messages
IMAP (Internet Message Access Protocol): Used for accessing and managing messages stored on the server

While POP3 also exists, this article focuses on SMTP and IMAP, which offer advantages for synchronization and email management across multiple devices.

📤 SMTP - Simple Mail Transfer Protocol

SMTP is the "engine" that sends emails. Here's a summary of what I learned:

Primary Function: SMTP is responsible for sending messages from the sender to the server, and between servers, ensuring the email reaches its recipient.
How It Works:
- The process begins with the email client sending the message to the SMTP server
- Commands like EHLO/HELO, MAIL FROM, RCPT TO, and DATA structure this communication
Ports and Security:
- Common ports include 25 (traditional) and 587 (for authenticated sending)
- Using TLS or SSL encryption is highly recommended to protect data during transmission

📥 IMAP - Internet Message Access Protocol

While SMTP handles sending, IMAP is the "manager" of received emails:

Primary Function: IMAP allows you to access and manage emails directly on the server. This means that regardless of which device you use—smartphone, tablet, or computer—your messages remain synchronized.
Advantages over POP3:
- Messages remain stored on the server, allowing viewing and organization from any device
- You can create, move, and delete folders directly on the server without downloading emails
How It Works:
- IMAP synchronizes message status (e.g., read/unread) and organizes emails into folders
Ports and Security:
- Port 143 is used for unencrypted connections, while port 993 is for secure connections via SSL/TLS

⚖️ Comparison Between SMTP and IMAP

Although both protocols are part of email communication, they have distinct, complementary roles:

SMTP:
- Responsible for sending emails
- Functions as the "digital mail carrier" distributing messages between servers
IMAP:
- Responsible for accessing and managing stored messages
- Allows you to track your emails in a synchronized manner across multiple devices

⚙️ Technical Aspects and Implementation

During my experience, I found that visualizing the email flow helps better understand how these protocols interact:

Sending and Receiving Flow:
- The email leaves the client and goes to the SMTP server
- From there, it may be forwarded to other servers until it reaches the recipient's server
- Finally, the email client uses IMAP to access and manage this message on the server
Configuration and Authentication:
- Correctly configuring ports and authentication methods (such as TLS/SSL) is essential to ensure everything works smoothly
- In many situations, the challenge lies precisely in aligning these configurations for efficient and secure customer service operation

🔒 Security and Best Practices

To avoid headaches (like the one I faced), it's important to pay attention to these points:

Common Threats:
- Without proper security, data can be intercepted during transmission
- Configuration errors can create vulnerabilities for spam or even more serious attacks
Security Measures:
- Always use encryption (SSL/TLS) in communications
- Regularly verify and update server authentication configurations
- Follow recommended best practices for both administrators and end users

✅ Conclusion

Understanding SMTP and IMAP protocols was essential to solving the problem I encountered when configuring the customer service system. With SMTP handling sending and IMAP enabling access and message management, both complement each other to make email communication fluid and secure.

I hope this article, written simply and based on my experience, helps you better understand how these protocols work. After all, sharing knowledge is a way to grow and make life easier for those who also need to deal with these configurations daily.

If you have questions or want to explore any topic in greater depth, feel free to comment. We're all learning together!

Prometheus + Redis: Simple Guide to Monitor Redis Instances

Lucas Aguiar — Mon, 21 Apr 2025 22:26:33 +0000

Redis is often a critical part of modern infrastructure — whether used as a cache, message broker, or ephemeral store. Monitoring it properly helps you detect issues early and ensure system stability.

In this guide, you’ll learn how to build a Redis monitoring dashboard with Prometheus and Grafana to quickly identify issues and improve system reliability.

⚙️ Installing Prometheus (Quick Setup)

We’ll start by deploying Prometheus using Helm:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

helm install kube-prometheus prometheus-community/kube-prometheus-stack \
  --namespace prometheus --create-namespace

This will install Prometheus, Grafana, and some built-in exporters.
To access Grafana:

kubectl port-forward svc/kube-prometheus-grafana -n prometheus 3000:80

You can access here -> http://localhost:3000/login

Default login: admin / prom-operator

🧱 Installing Redis

helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update

helm install redis bitnami/redis \
  --namespace app-stg \
  --set architecture=standalone \
  --set auth.enabled=false \
  --create-namespace

This installs a standalone Redis with no password, perfect for testing.
After this, you can point the Redis Exporter and your application to the correct Redis service. The hostname will be:

redis-master.app-stg.svc.cluster.local:6379

📦 Installing Redis Exporter

Now let’s deploy the Redis Exporter:

helm install redis-exporter prometheus-community/prometheus-redis-exporter \
  --namespace prometheus \
  --set serviceMonitor.enabled=true \
  --set serviceMonitor.namespace=prometheus \
  --set serviceMonitor.interval=15s \
  --set serviceMonitor.labels.release=kube-prometheus \
  --set redisAddress=redis://redis-master.app-stg.svc.cluster.local:6379 \
  --create-namespace

The Redis Exporter exposes metrics at a Prometheus-compatible /metrics endpoint.

kubectl port-forward svc/redis-exporter-prometheus-redis-exporter -n prometheus 9121:9121

Then open: http://localhost:9121/metrics
You should see a long list of metrics like this:

✅ redis_up 1 means the exporter is connected to a Redis instance and it’s healthy.
🚫 redis_up 0 means the exporter failed to connect — check the Redis address or service availability.

If everything is working, you’ll also see metrics like:

redis_connected_clients
redis_memory_used_bytes
redis_total_commands_processed

🔍 Checking in Prometheus

To validate that Prometheus is collecting Redis metrics:

kubectl port-forward svc/kube-prometheus-kube-prome-prometheus -n prometheus 9090

Open http://localhost:9090, go to Status > Targets, and check that Redis Exporter is listed and UP.

Everything is working right!

📊 Importing the Redis Dashboard on Grafana

To visualize your Redis metrics in a clean and ready-to-use panel, you can import a community-maintained dashboard from Grafana Labs.

https://grafana.com/grafana/dashboards/14091-redis-dashboard-for-prometheus-redis-exporter-1-x/

🧭 How to Import

Access Grafana by establishing port forwarding and navigating to http://localhost:3000 in your browser
Navigate to the "+" icon in the left sidebar and select "Import"
Enter the dashboard ID or URL in the provided field and click "Load"
Click "Import" to finalize the installation

Allow a few moments for the system to process your request as the metrics populate on your dashboard

Thats it!
⎈Happy Helming!⎈

Explorando um Cluster MongoDB com Terraform

Lucas Aguiar — Sun, 02 Feb 2025 02:12:41 +0000

A pouco tempo após algumas conversas me interessei por aprender mais sobre terraform e como provisionar infraestrutura de maneira mais fácil, simples de replicar em outros lugares e manter a longo prazo.

A ideia aqui é demonstrar como a abordagem de Infrastructure as Code pode simplificar a criação de um cluster MongoDB, composto por um nó primário e dois nós secundários. Utilizando o Terraform e o Docker como provedor para teste local.

O passo a passo

1. Preparando o ambiente

Primeiro, vamos definir o docker como provedor e uma rede para que os containers se comuniquem usando a resolução de DNS interno, podendo usar "mongo_primary", "mongo_secondary_1" e "mongo_secondary_2" apontando para os ips internos corretos.

terraform {
  required_providers {
    docker = {
      source  = "kreuzwerker/docker"
      version = "~> 3.0.2"
    }
  }
}

provider "docker" {}

resource "docker_network" "mongo_network" {
  name = "mongo_network"
}

2. Gerando os arquivos internos de autenticação

Antes de subir os containers, é necessário gerar um arquivo keyfile que será utilizado para a autenticação interna entre os nós do replica set do MongoDB. Esse arquivo garante que somente os nós autorizados possam se comunicar.

openssl rand -base64 756 > mongo-keyfile

Alterando as permissões para que somente o proprietário possa ler o arquivo

chmod 400 mongo-keyfile

Certifique-se de que o arquivo mongo-keyfile esteja na mesma pasta onde se encontra o arquivo main.tf ( ou em uma pasta no mesmo nível ), pois ele será referenciado nos volumes dos containers.

3. Configurando o replicaset

Separamos aqui os containers em duas partes 1 nó primário e 2 secundários, em um cluster replica set de mongodb o nó primário é responsável por aceitar operações de escrita e propagar as alterações para os secundários, que mantêm uma cópia dos dados e podem ser utilizados para balanceamento de carga em leituras.

// init-replica.js

conn = new Mongo("mongodb://admin:adminpass@localhost:27017");
db = conn.getDB("admin");

try {
    let status = rs.status();
    print("ℹ️ Replica Set já configurado.");
} catch (err) {
    if (err.codeName === "NotYetInitialized") {
        print("🚀 Iniciando Replica Set...");
        rs.initiate({
            _id: "rs0",
            members: [
                { _id: 0, host: "mongo_primary:27017" },
                { _id: 1, host: "mongo_secondary_1:27017" },
                { _id: 2, host: "mongo_secondary_2:27017" }
            ]
        });
        print("✅ Replica Set inicializado com sucesso!");
    } else {
        print("❌ Erro ao verificar status do Replica Set:", err);
        throw err;
    }
}

4. Subindo os containers mongodb

No terraform a configuração dos containers mongodb é similar, entretanto, só o nó primário recebe instruções de exposição das portas e o script de configuração do cluster.

Montamos os dois volumes referente ao keyfile de autenticação e o script de configuração de replica em js.

resource "docker_container" "mongo_primary" {
  name    = "mongo_primary"
  image   = "mongo:7.0"
  restart = "always"

  networks_advanced {
    name = docker_network.mongo_network.name
  }

  ports {
    internal = 27017
    external = 27017
  }

  volumes {
    host_path      = "${abspath(path.module)}/mongo-keyfile"
    container_path = "/data/keyfile"
  }

  volumes {
    host_path      = "${abspath(path.module)}/init-replica.js"
    container_path = "/docker-entrypoint-initdb.d/init-replica.js"
  }

  env = [
    "MONGO_INITDB_ROOT_USERNAME=admin",
    "MONGO_INITDB_ROOT_PASSWORD=adminpass",
    "MONGO_REPLICA_SET_NAME=rs0"
  ]

  command = [
    "mongod",
    "--replSet", "rs0",
    "--keyFile", "/data/keyfile",
    "--bind_ip_all"
  ]

  provisioner "local-exec" {
    command = "sleep 10 && docker exec mongo_primary mongosh /docker-entrypoint-initdb.d/init-replica.js"
  }
}

resource "docker_container" "mongo_secondary_1" {
  name  = "mongo_secondary_1"
  image = "mongo:7.0"
  restart = "always"

  networks_advanced {
    name = docker_network.mongo_network.name
  }

  volumes {
    host_path      = "${abspath(path.module)}/mongo-keyfile"
    container_path = "/data/keyfile"
  }

  env = [
    "MONGO_INITDB_ROOT_USERNAME=admin",
    "MONGO_INITDB_ROOT_PASSWORD=adminpass",
    "MONGO_REPLICA_SET_NAME=rs0"
  ]

  command = [
    "mongod",
    "--replSet", "rs0",
    "--keyFile", "/data/keyfile",
    "--bind_ip_all"
  ]
}

resource "docker_container" "mongo_secondary_2" {
  name  = "mongo_secondary_2"
  image = "mongo:7.0"
  restart = "always"

  networks_advanced {
    name = docker_network.mongo_network.name
  }

  volumes {
    host_path      = "${abspath(path.module)}/mongo-keyfile"
    container_path = "/data/keyfile"
  }

  env = [
    "MONGO_INITDB_ROOT_USERNAME=admin",
    "MONGO_INITDB_ROOT_PASSWORD=adminpass",
    "MONGO_REPLICA_SET_NAME=rs0"
  ]

  command = [
    "mongod",
    "--replSet", "rs0",
    "--keyFile", "/data/keyfile",
    "--bind_ip_all"
  ]
}

5. Integração com aplicações

Uma vez com o cluster configurado, é possível se conectar a ele através do comando:

docker exec -it mongo_primary mongosh "mongodb://admin:adminpass@localhost:27017/?replicaSet=rs0"

Para que uma aplicação tenha acesso ao cluster, é necessário que ela esteja junto a "mongo_network" para que não haja problemas na resolução de DNS.

Conclusão

Este exemplo demonstra como o terraform pode ser uma ferramenta poderosa para criar ambientes consistentes e fáceis de serem replicados, lembrando que o exemplo anterior não é recomendado para uso em produção pois cada nó deve estar em uma máquina separada para garantir alta disponibilidade.

Caso tenha ficado algo confuso, você pode acompanhar meu código no Github:

https://github.com/lusqua/terraform-mongodb-cluster

Desvendando a VPC na AWS: Subnets, Gateways e Configuração de Rede

Lucas Aguiar — Thu, 31 Oct 2024 10:30:00 +0000

Ontem à noite, após o expediente, decidi resolver alguns problemas técnicos em um side project na AWS. O desafio? Criar uma infraestrutura robusta, com subnets públicas e privadas, garantindo que apenas os recursos certos tivessem acesso à internet. A ideia era entender de uma vez por todas como cada peça se encaixa: Internet Gateway, NAT Gateway e os detalhes de roteamento.

O que parecia simples se revelou mais complexo. Ao configurar meu Auto Scaling Group, notei que as instâncias EC2 não estavam acessando a internet, mesmo quando atribuídas a subnets públicas. Isso me fez mergulhar fundo nas configurações de roteamento, nas permissões de IP público e nas nuances da AWS.

Aqui estão as lições e conceitos principais que consegui extrair desse sofrimento desenfreado:

Subnet Pública vs. Privada

Para simplificar, uma subnet pública permite que instâncias tenham acesso direto à internet através de um Internet Gateway (IGW), essencial para servidores web ou APIs externas. No entanto, minha aplicação precisava de segurança adicional, então configurei uma subnet privada para meus recursos internos, como bancos de dados. Essas subnets privadas não têm acesso direto à internet, mas, com um NAT Gateway, consegui permitir que os recursos façam atualizações ou se comuniquem externamente quando necessário.

Configurando o Internet Gateway e o NAT Gateway

Para as subnets públicas, configurei um Internet Gateway e adicionei uma rota que apontava 0.0.0.0/0 para ele, o que garantiu que as instâncias pudessem enviar e receber tráfego da internet. Já para as subnets privadas, configurei um NAT Gateway na subnet pública, permitindo que o tráfego externo fosse roteado indiretamente. Isso evitou a exposição direta à internet, mas ainda permitiu que as instâncias privadas acessem atualizações externas.

Desafios com o Auto Scaling e Distribuição de Tarefas

Um dos problemas mais complexos foi garantir que as tarefas do ECS fossem distribuídas entre as instâncias EC2 de maneira balanceada. Inicialmente, minhas tarefas não se espalhavam como esperado, então experimentei diferentes estratégias de Task Placement, como “AZ balanced binpack”, que ajudou a otimizar o uso das instâncias e garantir redundância entre as zonas de disponibilidade.

Após uma noite inteira, finalmente consegui configurar uma arquitetura estável, com recursos bem distribuídos e segura. Este side project foi uma experiência reveladora, me dando não apenas o domínio sobre subnets e gateways na AWS, mas também uma lição sobre resiliência e atenção aos detalhes.

Conclusão

O processo de configurar subnets públicas e privadas, Internet Gateway, e NAT Gateway pode ser complicado, mas é essencial para arquiteturas seguras e escaláveis. Para quem enfrenta os mesmos desafios, espero que este artigo forneça uma visão prática e inspiradora para perseverar, mesmo que isso signifique uma longa noite de trabalho.

Automatizando deployments com WatchTower e GitHub Container Registry

Lucas Aguiar — Tue, 08 Oct 2024 17:05:55 +0000

Manter aplicações atualizadas é crucial para garantir segurança, performance e a incorporação de novas funcionalidades. No mundo dos containers Docker, WatchTower surge como uma ferramenta poderosa para automatizar esse processo. Neste artigo, vamos explorar como integrar o WatchTower com o GitHub Container Registry (GHCR), criando um fluxo contínuo de atualização para seus containers. Vamos abordar desde a configuração inicial até um exemplo prático para ilustrar todo o processo.

🚀 Introdução ao WatchTower e GHCR

O que é o WatchTower?

WatchTower é uma ferramenta de código aberto que monitora seus containers Docker e automaticamente os atualiza quando uma nova imagem é detectada no registry configurado. Isso elimina a necessidade de intervenções manuais para manter suas aplicações sempre atualizadas com as últimas versões.

O que é o GitHub Container Registry (GHCR)?

O GitHub Container Registry é um serviço de registro de containers integrado ao GitHub, permitindo que você armazene e gerencie suas imagens Docker diretamente nos repositórios do GitHub. Ele oferece integração perfeita com workflows de CI/CD, facilitando a publicação e o gerenciamento de imagens.

🛠️ Pré-requisitos

Antes de começarmos, certifique-se de ter o seguinte:

Docker instalado na sua máquina ou servidor.
Conta no GitHub com acesso ao GitHub Container Registry.
Repositório GitHub para hospedar sua aplicação e workflow.
Conhecimentos básicos de Docker e GitHub Actions.

🔧 Passo a Passo: Integrando WatchTower com GHCR

Configurando o GitHub Container Registry

Primeiro, vamos configurar o GHCR para armazenar nossas imagens Docker.

a. Autenticação no GHCR

Para autenticar o Docker com o GHCR, você precisa de um token de acesso pessoal com permissão para write:packages e read:packages.

echo $CR_PAT | docker login ghcr.io -u USERNAME --password-stdin

Substitua CR_PAT pelo seu token de acesso pessoal e USERNAME pelo seu nome de usuário do GitHub.

b. Criando uma Imagem Docker

Vamos criar uma imagem Docker simples para este exemplo. Crie um arquivo Dockerfile:

# Dockerfile
FROM node:14-alpine
WORKDIR /app
COPY package.json .
RUN npm install
COPY . .
EXPOSE 3000
CMD ["npm", "start"]

E um arquivo package.json básico:

{
  "name": "watchtower-example",
  "version": "1.0.0",
  "main": "index.js",
  "scripts": {
    "start": "node index.js"
  },
  "dependencies": {
    "express": "^4.17.1"
  }
}

Crie também um index.js simples:

const express = require('express');
const app = express();
const port = 3000;

app.get('/', (req, res) => {
  res.send('Hello from WatchTower and GHCR!');
});

app.listen(port, () => {
  console.log(`App running on port ${port}`);
});

c. Construindo e Publicando a Imagem

Construa a imagem Docker e publique-a no GHCR.

docker build -t ghcr.io/USERNAME/watchtower-example:latest .
docker push ghcr.io/USERNAME/watchtower-example:latest

Substitua USERNAME pelo seu nome de usuário do GitHub.

Configurando o WatchTower

Agora, vamos configurar o WatchTower para monitorar o GHCR e atualizar automaticamente nossos containers.

a. Criando um Arquivo de Configuração do WatchTower

Crie um arquivo docker-compose.yml para orquestrar o WatchTower e sua aplicação.

version: '3'

services:
  app:
    image: ghcr.io/USERNAME/watchtower-example:latest
    container_name: watchtower-example
    ports:
      - "3000:3000"
    environment:
      - NODE_ENV=production

  watchtower:
    image: containrrr/watchtower
    container_name: watchtower
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    environment:
      - WATCHTOWER_CLEANUP=true
      - WATCHTOWER_POLL_INTERVAL=300
    command: --registry ghcr.io --interval 300

Explicações:

app: Serviço da aplicação que será monitorada pelo WatchTower.
watchtower: Serviço do WatchTower configurado para monitorar o GHCR com um intervalo de verificação de 5 minutos (300 segundos).

b. Variáveis de Ambiente para Autenticação

Para que o WatchTower possa acessar o GHCR privado, precisamos fornecer as credenciais. Atualize o serviço watchtower no docker-compose.yml:

environment:
  - WATCHTOWER_CLEANUP=true
  - WATCHTOWER_POLL_INTERVAL=300
  - WATCHTOWER_REGISTRY_AUTH=true
  - WATCHTOWER_REGISTRY_USERNAME=USERNAME
  - WATCHTOWER_REGISTRY_PASSWORD=CR_PAT

Substitua USERNAME pelo seu nome de usuário do GitHub e CR_PAT pelo seu token de acesso pessoal.

Automatizando a Atualização com GitHub Actions

Para garantir que sempre que você fizer push no repositório, a imagem Docker seja atualizada no GHCR, vamos configurar um workflow no GitHub Actions.

Crie um arquivo .github/workflows/docker-publish.yml no seu repositório:

name: Publish Docker image

on:
  push:
    branches: [ main ]

jobs:
  build-and-push:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v3

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Login to GHCR
        uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Build and push
        uses: docker/build-push-action@v4
        with:
          context: .
          push: true
          tags: ghcr.io/USERNAME/watchtower-example:latest

Explicações:

actions/checkout@v3: Faz o checkout do código.
docker/setup-buildx-action@v3: Configura o Docker Buildx.
docker/login-action@v3: Autentica no GHCR usando o token do GitHub.
docker/build-push-action@v4: Constrói e publica a imagem no GHCR.

Substitua USERNAME pelo seu nome de usuário do GitHub.

Implementando e Testando

a. Deploy Inicial

Inicie os serviços usando o Docker Compose:

docker-compose up -d

Isso iniciará sua aplicação e o WatchTower. O WatchTower começará a monitorar o GHCR a cada 5 minutos (300 segundos) para verificar se há novas imagens disponíveis.

b. Atualizando a Aplicação

Faça uma alteração no código da sua aplicação, por exemplo, atualize a mensagem em index.js:

res.send('Hello from WatchTower, GHCR, and your updated app!');

Commit e push as alterações para o repositório GitHub:

git add .
git commit -m "Update welcome message"
git push origin main

O GitHub Actions irá construir e publicar a nova imagem no GHCR. Após até 5 minutos, o WatchTower detectará a nova imagem e atualizará automaticamente o container watchtower-example sem downtime.

🎉 Conclusão

Integrar o WatchTower com o GitHub Container Registry oferece uma solução eficiente para manter suas aplicações Docker sempre atualizadas de forma automática. Essa automação não apenas economiza tempo, mas também garante que suas aplicações estejam sempre rodando as versões mais recentes e seguras. Além disso, combinando com ferramentas como o Traefik, você pode criar uma infraestrutura robusta e escalável, pronta para atender às demandas de ambientes de produção modernos.

Experimente implementar essa integração no seu projeto e aproveite os benefícios da automação contínua!

📚 Recursos Adicionais

Implementing DataLoader and Understanding Its Advantages Over Lookup

Lucas Aguiar — Thu, 15 Aug 2024 00:48:20 +0000

If you’ve been working with databases, especially in a GraphQL environment, you might have encountered the "N+1 problem." This issue occurs when your system makes multiple database calls to fetch related data, which can significantly slow down your application. Enter DataLoader—a tool that’s here to save the day by batching and caching database requests, making your queries much more efficient.

Let’s dive into how DataLoader works and see it in action with a practical example involving bank transactions and user details.

What is DataLoader?

DataLoader is a utility designed to batch and cache requests for data. Instead of making multiple calls to fetch each piece of data separately, DataLoader groups these requests together, sends them in one go, and then caches the results for future use. This approach is particularly useful in reducing the number of database queries, thus speeding up your application.

How DataLoader Solves the N+1 Problem

Consider a scenario where you have a list of banking transactions, and each transaction is associated with a user. Without DataLoader, you might end up querying the database for each user separately, leading to multiple (and often redundant) database queries. This is where DataLoader shines—it batches these requests together, so the database is queried only once per user.

Implementing DataLoader: A Practical Example

Let’s imagine you’re building a banking application, and you need to list transactions along with the details of the users who made those transactions. Here’s how you can implement DataLoader to optimize this process:

Setting Up the DataLoader:

   import DataLoader from 'dataloader';
   import { ObjectId } from 'mongodb';
   import { getCollection } from './mongodbConfig';  // Your MongoDB configuration file

   // Batch function to fetch users by IDs
   export const usersInBatch = async (ids) => {
     const coll = await getCollection('users');
     const objIdIds = ids.map(id => new ObjectId(id));
     const found = await coll.find({ _id: { $in: objIdIds } }).toArray();
     return found;
   };

   // Create a DataLoader instance for users
   export const userLoader = new DataLoader(usersInBatch);

Here, usersInBatch is a batch loading function that fetches user details for multiple userIds at once, and userLoader is the DataLoader instance that leverages this function.

Using DataLoader in Your Application:

   import { userLoader } from './dataLoaderConfig';  // Your DataLoader configuration file

   // Function to get transaction details
   export const getTransactionDetails = async (transaction) => {
     const user = await userLoader.load(transaction.userId);
     return {
       ...transaction,
       userName: user.name,
       userEmail: user.email,
     };
   };

   // Function to list transactions with user details
   export const listTransactions = async (page = 1, limit = 20) => {
     const coll = await getCollection('transactions');
     const skip = (page - 1) * limit;
     const rawTransactions = await coll.find().skip(skip).limit(limit).toArray();

     return await Promise.all(rawTransactions.map(getTransactionDetails));
   };

In this setup:

The getTransactionDetails function uses userLoader to fetch user details by userId, ensuring that each user is only fetched once per batch.
listTransactions retrieves a page of transactions and maps over them to enrich each transaction with the relevant user details.

Why Use DataLoader?

By using DataLoader, you reduce the number of database queries from potentially hundreds (or more) to just a few. This not only optimizes performance but also reduces load on your database, which can be crucial in high-traffic applications. Moreover, DataLoader’s built-in caching mechanism ensures that once a user’s data is fetched, it doesn’t need to be fetched again in the same request cycle, further speeding up your application.

Conclusion

DataLoader is an invaluable tool when dealing with scenarios that involve repeated database lookups, particularly in applications with complex data relationships like the one we explored here. By batching and caching requests, it dramatically improves the efficiency and performance of your database interactions. Whether you’re dealing with banking transactions, product orders, or any other domain where related data is frequently accessed, DataLoader can help you keep your application running smoothly and efficiently.

If you're facing performance issues due to repeated database queries, give DataLoader a try—your application (and your users) will thank you!

Forem: Lucas Aguiar

Understanding MongoDB $lookup performance

The problem

Testing

Understanding the lookup

Understanding the problem of $expr

Improving the pipeline even more

Conclusion

Understanding IMAP and SMTP Mail Protocols

📮 Introduction

🌐 Overview of Email Protocols

📤 SMTP - Simple Mail Transfer Protocol

📥 IMAP - Internet Message Access Protocol

⚖️ Comparison Between SMTP and IMAP

⚙️ Technical Aspects and Implementation

🔒 Security and Best Practices

✅ Conclusion

Prometheus + Redis: Simple Guide to Monitor Redis Instances

⚙️ Installing Prometheus (Quick Setup)

🧱 Installing Redis

📦 Installing Redis Exporter

🔍 Checking in Prometheus

📊 Importing the Redis Dashboard on Grafana

🧭 How to Import

Explorando um Cluster MongoDB com Terraform

O passo a passo

1. Preparando o ambiente

2. Gerando os arquivos internos de autenticação

3. Configurando o replicaset

4. Subindo os containers mongodb

5. Integração com aplicações

Conclusão

Desvendando a VPC na AWS: Subnets, Gateways e Configuração de Rede

Subnet Pública vs. Privada

Configurando o Internet Gateway e o NAT Gateway

Desafios com o Auto Scaling e Distribuição de Tarefas

Conclusão

Automatizando deployments com WatchTower e GitHub Container Registry

🚀 Introdução ao WatchTower e GHCR

🛠️ Pré-requisitos

Implementing DataLoader and Understanding Its Advantages Over Lookup

What is DataLoader?

How DataLoader Solves the N+1 Problem

Implementing DataLoader: A Practical Example

Why Use DataLoader?

Conclusion

Understanding the problem of `$expr`