Forem: Jose Luis

How to install n8n self-hosted on AWS

Jose Luis — Wed, 08 Apr 2026 15:51:52 +0000

In this post, you will install and configure your own n8n instance using AWS cloud services. This gives you more control over infrastructure, costs, and workflow customization.

Using n8n in your own infrastructure (self-hosted) is an alternative to automate workflows without paying a monthly subscription to n8n.io cloud services.

What Will You Learn?

In this post, you will learn how to:

Install and configure n8n as self-hosted
Estimate basic operation costs

Step-by-Step Implementation

1. Prerequisites ✅

Before you start, you need:

An active Amazon Web Services (AWS) account
A domain to publish n8n with HTTPS
Download the whatsapp-chatbot repository for installation and configuration of n8n

2. Deploy n8n on AWS 🚀

The installation is automated using a CloudFormation template that creates:

AWS Lightsail instance
Network and communication permissions
Static public IP address
Startup script to install docker, n8n, database (postgres), web server (nginx), and certbot

Template Parameters:

InstanceName: instance name
BlueprintId: base image, for example amazon_linux_2023
BundleId: instance size
AvailabilityZone: AWS region and zone
StaticIPName: static IP name
DBUser: database user
DBPassword: database password
Domain: public domain for n8n
Email: email for HTTPS certificate

You can run the CloudFormation template using AWS CLI or AWS Console.

Option A: Deploy from Command Line 💻

Configure access keys to connect AWS CLI with your AWS account. Check details here.
Go to the path where you downloaded the whatsapp-chatbot project, inside the aws folder
Run this command. Replace Domain and Email with your values:

aws cloudformation deploy \
    --stack-name n8n \
    --template-file template.yaml \
    --region us-east-1 \
    --parameter-overrides \
        InstanceName=n8n-instance \
        AvailabilityZone=us-east-1a \
        BlueprintId=amazon_linux_2023 \
        BundleId=small_3_0 \
        Domain={Domain} \
        DBUser=admin \
        DBPassword=admin \
        Email={email} \
        StaticIpName=n8n-ip-address

Option B: Deploy from AWS Console 🌐

Log in to AWS Console
Go to CloudFormation and create a new stack
Import the template (template.yaml) and follow the steps to set parameters like Domain and Email

3. Validate the installation 📝✅

After CloudFormation finishes, check:

The Lightsail instance exists
Security rules allow SSH, HTTP, and HTTPS
The static IP is assigned
You can connect using SSH
The n8n and postgres containers are running

It is recommended to restrict SSH access only to your IP.

4. Configure HTTPS 🔐

This step is required so n8n can connect with platforms like WhatsApp, Google, etc., that require SSL/TLS.

Steps:

Create the DNS zone
Point the domain to the Lightsail static IP
Generate the certificate with Certbot
Verify that n8n works with HTTPS
Register in the n8n interface

Connect to your Lightsail instance using SSH and run:

sudo certbot --nginx -d n8n.tu-dominio.com \
  --email tu-correo@dominio.com \
  --agree-tos \
  --no-eff-email \
  --redirect

Licensing 📄

n8n uses the Fair Code license (Sustainable Use License). You can use it for free, modify it, and run it in your own infrastructure (self-hosted) for personal or business use.

You can use it inside internal products or services in a company, as long as n8n is not the main product.

You cannot resell n8n as a hosted automation platform or create a business where the main product is a managed n8n service.

Costs 💰

AWS Lightsail (Self Hosting) ☁️

In AWS Lightsail, the minimum recommended instance to run n8n is:

2 GB RAM
2 vCPU
60 GB SSD

This setup supports basic workflows with good stability. This instance has a monthly cost.

Conclusions

⚖️ You can use n8n in internal or business projects, but you cannot resell it as the main service.

💰 n8n is free if you use your own infrastructure (self-hosted), but AWS infrastructure has monthly costs.

🚀 Infrastructure and n8n installation are automated using AWS CloudFormation (Infrastructure as Code). You can deploy n8n in a few steps.

How to build a WhatsApp Chatbot with n8n, AWS, and OpenAI

Jose Luis — Tue, 07 Apr 2026 16:08:48 +0000

If you want to build your own WhatsApp chatbot to receive text and voice messages, process them with AI, and reply automatically, this guide shows a practical way to do it using n8n + OpenAI + Meta + AWS.

🧠 What will you learn?

In this post you will see how to:

Connect OpenAI AI services with n8n
Connect WhatsApp Business Cloud services with n8n
Receive text and voice messages and reply automatically from an n8n workflow
Estimate basic operation costs

🛠️ Implementation step by step

1. Prerequisites ✅

Before you start, you need:

An OpenAI account with balance
A Meta Developer account (you can use your personal Facebook account)
A domain to publish n8n with HTTPS
Access to n8n (self-hosted or cloud service)
Download the whatsapp-chatbot repository for installation and setup

2. Import chatbot workflow 💬

Import the workflow from the n8n folder in the whatsapp-chatbot repository. Copy and paste the content of the whatsapp-chatbot.json file into n8n.

This workflow allows you to:

Receive text or voice messages from WhatsApp
Detect if the content is audio
Download and transcribe the audio
Run an OpenAI LLM model
Send the response back to WhatsApp

3. Configure OpenAI in n8n 🧠

To run OpenAI Large Language Models (LLM) in n8n, you need an API Key to authenticate requests. Steps:

Go to OpenAI Platform with your account
In the side panel, open API Key and create a key
In the n8n workflow, in the Transcribe Recording node, configure OpenAI credentials

4. WhatsApp Business API 📲

To send and receive voice and text messages from WhatsApp, you need to create an app in Meta Developer and configure it to get credentials for n8n.

Steps:

Create an app in the Meta Developers portal
Add the use case Connect with customers through WhatsApp
Confirm app creation

Send messages to WhatsApp 📤

To enable sending messages from n8n, you need authentication credentials. Steps:

In Meta Developer, open your app and go to the WhatsApp configuration. Copy the WhatsApp Business Account ID
In Meta Business, go to system users. Select or create a user
Assign the Meta Developer app to that system user
Generate an access token. Set permissions to whatsapp_business_messaging
In n8n workflow, in the Send Message (WhatsApp Business Cloud) node, create WhatsApp credentials using values from steps 1 and 4. Test the connection

Receive messages from WhatsApp 📥

We will use WhatsApp test numbers to receive messages. For real numbers, follow extra steps.

Each message triggers a Webhook in n8n. Steps:

In Meta Developer, go to app settings and get App ID and App Secret
In n8n workflow, in WhatsApp Business on Message node, create OAuth credentials with App ID and App Secret. Test connection
Publish the n8n workflow so it is available online
Copy the Production POST URL from the node (this is your webhook endpoint)
In Meta Developer, set this URL in the WhatsApp configuration with a verification token. Test connection

5. Test the chatbot 🤖

Use WhatsApp test phone numbers to run the chatbot workflow. Steps:

In Meta Developer, go to WhatsApp API settings and copy the test number
Send a message from your WhatsApp to that number

Costs 💰

OpenAI API 🤖

The cost of OpenAI API depends on:

Model used
Number of tokens
Input vs output usage

Output is usually 3–6 times more expensive than input.

Model	Input ($/1M tokens)	Output ($/1M tokens)	Context	Recommended use
GPT-5.4	$2.50	$15.00	~1M tokens	High complexity, agents
GPT-5.4 mini	$0.75	$4.50	400K tokens	Apps, SaaS, chatbots
GPT-5.4 nano	$0.20	$1.25	~400K tokens	Classification, extraction

Monthly estimate 📊

WhatsApp conversation (1–5 minutes):

Messages per conversation: 6–12
Words per message: 8–15
Total words: ~120

Conversion:

1 token ≈ 0.75 words
120 words ≈ 160 tokens

Type	Tokens
User input	80
Model output	80
Total	160

Monthly scenario 📅

Example: 10,000 conversations/month

Model	Monthly cost
GPT-5.4	$14 USD
GPT-5.4 mini	$4.2 USD
GPT-5.4 nano	$0.4 USD

Meta API 📱

WhatsApp Business API uses a pricing model based on message categories:

Service
Utility
Authentication
Marketing

Service messages (user starts the chat) are free within a 24-hour window.

Monthly estimate 📈

600 service messages → $0

Total Meta cost 💵: $0/month

n8n

n8n cost depends on infrastructure and number of workflow executions (conversations). Below table shows the comparation between Self-hosted using AWS cloud services and Cloud Starter:

Variable	Self-hosted (AWS)	Cloud Starter	Cloud Pro
Base cost	Low (infra + ops)	Medium (~€20–€30/mo)	High (~€50–€100+/mo)
Infrastructure	Your responsibility	Included	Included
Executions	Unlimited (depends infra)	Limited (~5k–10k/mo)	Higher (~20k–50k+/mo)
Scalability	Manual	Auto (limited)	Auto (better)
Security	Full control	Managed by n8n	Managed by n8n
Concurrency	Depends on resources	Limited	Higher
Integrations	All	All	All
Maintenance	High (needs DevOps)	Low	Low
Multi-user	Manual setup	Limited	Full (roles, teams)

Cómo crear un chatbot de WhatsApp con n8n, aws y OpenAI

Jose Luis — Tue, 31 Mar 2026 20:27:48 +0000

Si quieres crear tu propio chatbot de WhatsApp para recibir mensajes de texto y voz, procesarlos con IA y responder automáticamente, esta guía te muestra una forma práctica de hacerlo con n8n + OpenAI + Meta + AWS.

🧠 ¿Qué vas a aprender?

En este post vas a ver cómo:

Conectar los servicios de IA de OpenAI con n8n
Conectar los servicios de WhatsApp Business Cloud con n8n
Recibir mensajes de texto, voz y responder automáticamente desde un workflow de n8n.
Estimar costos básicos de operación

🛠️ Implementación paso a paso

1. Prerrequisitos ✅

Antes de empezar, necesitas:

Una cuenta de OpenAI con saldo
Una cuenta de Meta Developer. Puedes usar tu cuenta personal de Facebook.
Un dominio para publicar n8n con HTTPS
Acceso a n8n utilizando la instalación self-hosted o servicio en la nube.
Descargar el repositorio whatsapp-chatbot para la instalación y configuración del chatbot.

2. Importa chatbot workflow 💬

Importar el flujo de proceso (workflow) que se encuentra en la carpeta n8n del repositorio whatsapp-chatbot. Copia y pega el contenido del archivo whatsapp-chatbot.json en n8n. Este proceso permite:

Recibir mensajes de texto o voz desde WhatsApp
Identificar si el contenido es audio
Descargar el audio y transcribirlo
Ejecutar modelo de LLM de OpenAI
Envíar la respuesta nuevamente a WhatsApp

3. Configurar OpenAI en n8n 🧠

Para ejecutar los modelos Large Language Model (LLM) de OpenAI en n8n se necesita una API Key para autenticar las solicitudes realizadas desde n8n. Los pasos son:

Ingresar a Platform OpenAPI con tu cuenta
En el panel lateral ingresar a API Key para crear la llave
En el flujo de proceso de n8n, con el nodo de Transcribe Recording, configurar las credenciales OpenAI account.

4. WhatsApp Business API 📲

Para recibir y enviar mensajes de voz y texto desde WhatsApp se debe crear una aplicación en META Developer y configurarla para obtener las credenciales necesarias que permitan la conexión desde n8n. Los pasos son:

Crear una aplicación en el portal META Developers.
Añadir caso de uso Connect with customers through WhatsApp.
Confirmar la creación de la aplicación.

Enviar mensajes hacia WhastApp 📤

Para habilitar el envío de mensajes a WhatsApp desde n8n se requieren credenciales de autenticación para que WhatsApp acepte las solicitudes realizadas desde n8n. Los pasos son:

En Meta Developer, ingresar a la personalización del caso de uso de Connect with customers through WhatsApp de la aplicación, en la sección de de configuración de la API, copiar Identificador de la cuenta de WhatsApp Business
En META Business, con el portafolio empresarial usado en la creación de la aplicación, opción usuario del sistema, seleccionar un usuario existente o crear uno nuevo.
Asignar la aplicación de Meta Developer como activo al usuario del sistema.
Generar un identificador para la aplicación seleccionando una vigencia del identificador y la asignación de permisos que deben ser whatsapp_business_messaging.
En el flujo de proceso de n8n, con el nodo de Send Message de WhatsApp Business Cloud, crear las credenciales de WhatsApp account, copiando los valores de los puntos 1 y 4, y probar la conexión.

Recibir mensajes desde WhatsApp 📥

Vamos a usar los números de pruebas de WhatsApp para recibir los mensajes, para usar números propios seguir los siguientes pasos. Cada mensaje recibido desde el número configurado en la aplicación ejecuta un WebHook (Servicio Web de n8n) para notificar la recepción del mensaje y el contenido del mismo. Los pasos para la obtención de la credenciales y la configuración del WebHook son:

En Meta Developer, ingresar a la información Básica de la Aplicación en la sección de Configuración para consultar el App ID y App Secret.
En el flujo de proceso de n8n, con el nodo de WhatsApp Business on Message , crear credenciales WhatsApp OAuth account, con los valores del App ID y App Secret, posteriormente probar la conexión.
Publicar el flujo de proceso de n8n, para que estén expuestos sus servicios en internet.
En el flujo de proceso de n8n, con el nodo de WhatsApp Business on Message de n8n, copiar la URL Post de Producción que es el Servicio Web expuesto por tu instalación de n8n para ser invocado cada vez que se recibe un mensaje en WhatsApp.
En Meta Developer, ingresar a la personalización del caso de uso de Connect with customers through WhatsApp de la aplicación, en la sección de configuración, e ingresa la URL del punto 4 con un identificador de verificación. Posterior probar la conexión.

5. Probar el chatbot 🤖

Utilizando los números de celular de WhatsApp de pruebas podemos ejecutar el flujo de proceso de WhatsApp Chat Bot. Los pasos son:

En META Developer, ingresar a personalizar el Casos Uso WhatsApp de la aplicación, en la opción Configuración de la API, copiar el número de prueba.
Ingresar a tu WhatsApp y enviar mensaje al número del punto 1.

Costos 💰

OpenAI API 🤖

El costo de usar la API de OpenAI depende principalmente de tres variables: el modelo elegido, la cantidad de tokens procesados y el tipo de uso (entrada y salida). El costo se calcula por cada millón de tokens (texto procesado), donde el output (respuesta del modelo) suele ser entre 3 y 6 veces más caro que el input (texto enviado).

Modelo	Input ($/1M tokens)	Output ($/1M tokens)	Contexto	Uso recomendado
GPT-5.4	$2.50	$15.00	~1M tokens	Alta complejidad, agentes
GPT-5.4 mini	$0.75	$4.50	400K tokens	Apps, SaaS, chatbots
GPT-5.4 nano	$0.20	$1.25	~400K tokens	Diseñado para: clasificación extracción pipelines masivos

Cálculo mensual estimado 📊:

Conversación en WhatsApp de 1–5 min

Mensajes por conversación: 6–12
Palabras por mensaje: 8–15
Promedio total palabras: ≈ 120 palabras

Conversión estándar:

1 token ≈ 0.75 palabras
120 palabras ≈ 160 tokens

Tipo	Tokens
Input usuario	80 tokens
Output modelo	80 tokens
Total	160 tokens

Escenario mensual 📅

Supón: 10,000 conversaciones / mes

Modelo	Costo mensual
GPT-5.4	$14 USD
GPT-5.4 mini	$4.2 USD
GPT-5.4 nano	$0.4 USD

Meta API 📱

El modelo de costos de WhatsApp Business API funciona principalmente bajo un esquema de mensajes por categoría, donde el cobro depende del tipo de interacción y de quién inicia la conversación. Existen cuatro categorías: servicio, utilidad, autenticación y marketing. Los mensajes de servicio (cuando el cliente escribe primero) son gratuitos y permiten responder libremente dentro de una ventana de 24 horas con texto, audio, imágenes o documentos.

Cálculo mensual estimado 📈

600 mensajes de servicio iniciados por clientes → USD 0

Total Meta estimado 💵: USD 0/mes

n8n

Los costo de n8n dependen del la infraestructura y el número de ejecuciones del flujo del proceso, es decir en el número de conversaciones que se reciban en el chatbot. La comparación de opciones de uso de n8n con infraestructura en aws o de n8n.io es:

Variable	Self-hosted (AWS)	Cloud Starter	Cloud Pro
💰 Costo base	Bajo (infra + ops)	Medio (~€20–€30/mes)	Alto (~€50–€100+/mes)
⚙️ Infraestructura	Tu responsabilidad (AWS, VPS, etc.)	Incluida	Incluida
🔁 Ejecuciones (workflows)	Ilimitadas (según tu infra)	Limitadas (~5k–10k/mes)	Más altas (~20k–50k+/mes)
🚀 Escalabilidad	Manual	Automática (limitada)	Automática (mejor)
🔐 Seguridad	Total control	Gestionada por n8n	Gestionada por n8n
🔄 Concurrencia	Depende de recursos	Limitada	Mayor concurrencia
🔌 Integraciones	Todas	Todas	Todas
🛠️ Mantenimiento	Alto (requiere DevOps)	Bajo	Bajo
👥 Multiusuario	Configurable manual	Limitado	Completo (roles y equipos)

Cómo instalar tu propia instancia de n8n con AWS

Jose Luis — Tue, 31 Mar 2026 18:00:03 +0000

En este post vas a instalar y configurar tu propia instancia de n8n con los servicios de infraestructura en la nube de AWS para tener más control sobre la infraestructura, costos y personalización de workflows.

Utilizar n8n en tu propia infraestructura (self-hosted) es una alternativa para la automatización de flujos de trabajo sin pagar subscripción mensual en los servicios de n8n.io en la nube que gestionan la infraestructura por ti.

🧠 ¿Qué vas a aprender?

En este post vas a ver cómo:

Instalar y configurar n8n con self-hosted
Estimar costos básicos de operación

🛠️ Implementación paso a paso

1. Prerrequisitos ✅

Antes de empezar, necesitas:

Una cuenta activa de Amazon Web Services (AWS)
Un dominio para publicar n8n con HTTPS
Descargar el repositorio whatsapp-chatbot para la instalación y configuración de n8n.

2. Despliegue de n8n en AWS 🚀

La instalación se automatiza con una plantilla de CloudFormation que aprovisiona:

Instancia de AWS Lightsail
Permisos de red y comunicaciones
Dirección IP pública estática
Script de arranque para instalar docker, n8n, base de datos (postgres), servidor web (nginx), y certbot.

Parámetros la plantilla:

InstanceName: nombre de la instancia
BlueprintId: imagen base, por ejemplo amazon_linux_2023
BundleId: tipo/tamaño de la instancia
AvailabilityZone: región y zona de disponibilidad de aws.
StaticIPName: nombre de la IP estática
DBUser: usuario de la base de datos
DBPassword: password de la base de datos
Domain: dominio público para n8n
Email: correo para la generación del certificado HTTPS

La ejecución de la plantilla de Cloudformation se puede realizar desde línea de comandos con AWS CLI o por medio de la consola web de AWS.

Opción A: despliegue por línea de comandos 💻

Configurar las accesskey para la conexión de aws cli con tu cuenta de aws. Ver detalles aquí.
Ubícate en el path donde descargaste el proyecto whatsapp-chatbot en la carpeta aws.
Ejecutar el siguiente comando, modificando los valores de los parámetros Domain y Email que correspondan al dominio y correo electrónico de tu propiedad:

aws cloudformation deploy \
    --stack-name n8n \
    --template-file template.yaml \
    --region us-east-1 \
    --parameter-overrides \
        InstanceName=n8n-instance \
        AvailabilityZone=us-east-1a \
        BlueprintId=amazon_linux_2023 \
        BundleId=small_3_0 \
        Domain={Domain} \
        DBUser=admin \
        DBPassword=admin \
        Email={email} \
        StaticIpName=n8n-ip-address

Opción B: despliegue desde la consola web 🌐

Ingresar a la consola web de AWS
con tu cuenta.
Ir al servicio de Cloudformation de tu cuenta de AWS y crear stack con nuevos recursos.
Importar la planilla de Cloudformation (template.yaml) y seguir los pasos del wizard para ingresar el nombre del stack y los valores de los parámetros Domain y Email.

3. Validación de la instalación

Una vez finalizada la ejecución de la plantilla de Cloudformation, revisa:

Que la instancia de Lightsail exista
Que el grupo de acceso permita SSH, HTTP y HTTPS
Que la IP estática esté asignada
Que puedas entrar por SSH
Que los contenedores de n8n y postgres estén activos

También conviene restringir el acceso SSH para que solo esté permitido desde tu IP.

4. Configuración de HTTPS 🔐

Este paso es obligatorio para habilitar la conexión de n8n con plataformas como WhatsApp, Google, etc que requieren una conexión segura utilizando SSL/TLS. Los pasos son:

Crear la zona DNS
Asociar el dominio a la IP estática de Lightsail
Generar el certificado con Certbot
Validar que n8n abra por HTTPS
Registrarte en la interfaz de n8n

Accede a la instancia de Lightsail por medio de SSH y ejecuta el siguiente comando de Certbot para la generación de certificado SSL/TLS:

sudo certbot --nginx -d n8n.tu-dominio.com \
  --email tu-correo@dominio.com \
  --agree-tos \
  --no-eff-email \
  --redirect

Licenciamiento 📄

n8n se distribuye bajo la licencia Fair Code (Sustainable Use License). Esta licencia permite usar el software libremente, modificarlo y ejecutarlo en instalaciones propias (self-hosted) sin costo, tanto para uso personal como empresarial. Está permitido usarlo dentro de productos o servicios internos de una empresa, siempre que n8n no sea el producto principal ofrecido al cliente.

No se puede revender n8n como plataforma de automatización hospedada ni crear un negocio cuyo producto principal sea una instancia gestionada de n8n. Se restringe la reventa o provisión pública del software como servicio.

Costos 💰

AWS Lightsail (Self Hosting) ☁️

En AWS Lightsail, la instancia mínima razonable para operar n8n es la equivalente a 2 GB RAM / 2 vCPU / 60 GB SSD, que permite ejecutar workflows básicos con estabilidad. El precio por mes de este tipo y tamaño de instancia es:

Conclusiones

⚖️ Puedes usar n8n libremente en proyectos internos o empresariales, pero no puedes revenderlo como servicio principal. Esto limita modelos de negocio tipo SaaS basados en n8n.

💰 n8n es gratis cuando usas tu propia infraestructura (self-hosted), pero tu infraestructura con AWS tiene costos mensuales.

🚀 El aprovisionamiento de la infraestructura e instalación de n8n está automatizada con el servicio CloudFormation de infraestructura como código de AWS. Puedes usar n8n a distancia de clicks.

Data Streaming Architecture

Jose Luis — Wed, 27 Mar 2024 03:19:57 +0000

In a previous post, we studied the data streaming architecture basics, now we are going to set up AWS services to enable speed layer capabilities to ingest, aggregate, and store the streaming data. Each AWS service belongs to one or many stages of the data pipeline depending on their capabilities.

Kinesis Data Stream

Kinesis Data Stream is a serverless service with high throughput to ingest fast and continuous streaming data in real-time, it uses a shard to receive and store temporarily the data record in a unique sequence. A shard can support up to 1,000 RPS or 1 MB/sec writes and 2,000 RPS or 2 MB/sec read operations. The number of shards depends on the amount of data ingested and the level of throughput needed, more details about the capacity are here.

Go to Kinesis services and create a data stream using the on-demand capacity.

awsmeter JMeter Plugin

To produce streaming data we are going to use awsmeter, it is a JMeter plugin that uses AWS SDK + KPL (Kinesis Producer Library) to publish messaging on shards. You need an AWS access key and a data stream name. Find more details here.

The structure of this example message is using CloudEvents specification:

{
   "specversion" : "1.0",
    "type" : "kinesis-data-stream",
    "source" : "https://github.com/JoseLuisSR/awsmeter",
    "subject" : "event-created",
    "id" : "${eventId}",
    "time" : "${time}",
    "datacontentcoding" : "text/xml",
    "datacontenttype" : "text/xml",
    "data" : "<much wow=\"xml\"/>"
}

S3

Use S3 to store streaming data and query the data with S3 Select. It is integrated natively with Kinesis Firehose and is a fully managed and regional service. Create a S3 bucket.

Kinesis Firehose

It is a streaming ETL solution to capture, transform, and load stream data into AWS data stores. Kinesis Firehose is a serverless service, fully managed by AWS, and automatically scales to support the throughput you need. To create a delivery stream you choose the source and destination.

For source choose Kinesis Data Stream, in the source setting section search the kinesis data stream you created.

For Destination choose S3, in the destination setting section search S3 bucket you created.

Enable Dynamic partitioning for efficient query optimization.

Enable Inline parsing for JSON to use the Kinesis Data Firehose built-in support mechanism, a jq parser, for extracting the keys from messages for partitioning data records that are in JSON format. Specify the key name and JQ expression as below:

In the S3 bucket prefix section, choose Apply dynamic partitioning keys to generate partition key expressions. To enable Hive-compatible style partitioning by type and source, update the S3 bucket prefix default value with type= and source=.

In the S3 bucket error output prefix box, enter kdferror/. It will contain all the records that the Kinesis Data Firehose is not able to deliver to the specified S3 destination.

In the delivery stream setting section, expand the Buffer hints, compression, and encryption section.

For Buffer size, enter 64.
For Buffer interval, enter 60.

Kinesis Data Firehose buffers incoming streaming data to a certain size and for a certain period before delivering it to the specified destinations (S3). For a delivery stream where data partitioning is enabled, the buffer size ranges from 64 to 128MB, with the default set to 128MB, and the buffer interval ranges from 60 seconds to 900 seconds.

To Kinesis Firehose can access S3 need to use IAM roles with the permissions needed, on the advance settings choose existing IAM role

Review the stream configuration, and then choose Create delivery stream

S3 Select

Check your S3 Bucket to see if the data is stored on the folders of type and subject that are the dynamic partitions we configured. Choose one file, go to actions, and select the Query with S3 Select option.

Conclusion

In this post you have provisioned and set up aws services to enable a data streaming architecture solution, you have done the following:

Created Kinesis Data Stream.
Set up awsmeter to generate streaming messages.
Created a Kinesis Data Firehose stream and connected the Kinesis data stream to it.
Configured dynamic partitioning on the Kinesis Data Firehose delivery stream.
Delivered data to Amazon S3.

Data Streaming Architecture Basics

Jose Luis — Wed, 27 Mar 2024 03:19:24 +0000

In this post, we will understand the 6'V of Big Data, review the Data Pipeline and Lambda architecture to understand the complexity of getting, storing, and processing the data, and then set up AWS services to ingest and store streaming data to perform real-time analytics. Let's start.

6'V of Big Data

The amount of data, the speed of produced data, and the diversity of data are common in the systems today, data is everywhere and we need to choose the right data sources to get the most accurate and valuable data. The 6V’s of Big Data shows us the challenges we need to face when creating Data Streaming Architecture with cost optimization and performance efficiency:

Volume: Gigabytes, Terabytes, Petabytes, and more are the amount of data we need to receive, store, and process.
Velocity: The data is produced all day, Eg 150K videos are uploaded to YouTube every minute, and 66K photos are shared on Instagram every minute. The data is coming at scheduled hours (batch), real-time (streaming), or both.
Variety: The data are files, fields, text, audio, video, and images. The data is structured, semi-structured, and unstructured.
Veracity: With many data sources the data could be incomplete, outdated, repeated, or not real, the quality of the data is essential to get the right insights.
Variability: The data could change, depending on the seasons, geopolitics events, or just add a new data source. You need to validate how often the structure or shape of your data changes and the side effect is the meaning the data changes also.
Value: The main purpose is to give value, describing what is happening, what could happen, and the opportunities identified. The data insights enable organizations to become data-driven.

Data Pipeline

It is the building block for analytics solutions, it defines the layers and capabilities needed to optimize the ingestion, transformation, and storage of your analytics system. The layers are:

Data sources: Include databases, system logs, IoT signals, tape disks, and other kinds of storage systems with data related to your business. The data is structured, semi-structured, and unstructured.
Ingestion, move the data from external data sources to another location using tools like ETL, SDK, or middlewares, the tools depend on data type and workload requirements.
Storage, the data is stored temporarily or persistently in databases or object storage. The data is stored with format, partitioning, and compression for efficient storage and optimized querying.
Processing, to get performance efficiency and cost optimization, the data is cataloged for indexing and search, then processed to clean, complete, anonymization and enrich, and finally control access to enable confidentiality and integrity of the data.
Analytics and visualization provide descriptive and predictive analytics for discovering patterns and insights in data. This stage provides business decision-makers with graphical representations of analysis, making it easier to see the implications of the data.

Lambda architecture

It is not a lambda function, it is a data-processing architecture designed to handle massive quantities of data coming from batch and real-time streams, and serving the data for user queries. This is a layered architecture to distribute the responsibilities and load to get better latency, throughput, and fault tolerance, the layers are.

Batch, all the data sources with historical data or transaction information with restrictions to get data in real-time are in this layer with ETL and execute a map-reduce programming model to filter and sort information and reduce (summarize) the data using a distributed and parallel system.
Speed, most recent information like events that occurred, online transactions, and streaming data are in this layer with capabilities to aggregate, partition, and compress the data in near real-time. The data could not be accurate or complete as a batch layer but it is available almost immediately.
Serving, the information from batch and speed layers are joined and stored on this layer to enable analytics tools to do predictive and prescriptive analytics executing queries over precomputed views.

In the next post, we will set up Kinesis Data Stream + Kinesis Firehose + S3 to ingest and store streaming data to perform real-time analytics.

DynamoDB Basics, Part 1

Jose Luis — Mon, 27 Mar 2023 01:26:48 +0000

Along my journey to the cloud, I have discovered amazing services, one of them is DynamoDB. It is NoSQL Database and serverless service with capabilities to store information semi-structured or unstructured with high performance, availability, and scalability.

DynamoDB is an OLTP database that is focused on transaction-oriented tasks, today many of the applications or projects we are working on are to automatize tasks and transactional processing where need CRUD operations over a high volume of data, DynamoDB fits very well for these business cases, and you can use it instead SQL databases. I invite you to read Alex DeBrie's post about The What, Why, and When of Single-Table Design with DynamoDB to know in which cases you can use or not DynamoDB.

One of the big differences between DynamoDB and traditional SQL database is the number of tables, with one single table of DynamoDB you can handle any data model with entities, relationships (1:N, N:N, etc), and attributes then the queries in a database is different also, DynamoDB is a Key-Value database, to request information you should use a partition key, the partition determines the physical storage where data is. We need to learn a new way to design a data model for DynamoDB but before seeing the methodology we can start with the basics of DynamoDB.

When we face new technologies, we need to understand the problem solved, capabilities, boundaries, elements, and the reason why were created. I’m going to use some SQL database concepts to explain DynamoDB because there are similar things, and we can learn more easily by linking new knowledge, let’s start.

Table

Like an SQL database table, DynamoDB table is to store information. AWS replicates the data of the table in different AZ of the region, you can set up a global table to replicate the information in another AWS region. You can create many DynamoDB tables, but you don't need a table per entity as SQL database does, one single DynamoDB table can handle all the entities and relations in a single table.

Primary Key & Attributes

It should be unique and it is the value to identify the record in the table. It could be a single key with the Partition Key (PK) or a composite key with Partition Key + Sort Key (SK). When using a composite primary key you can share the PK in multiple items but the SK should be different in each record.

An attribute is a data element with type and value to store single or composite information like a list or map with data structures like JSON. DynamoDB is schema-less so each record could have different attributes. There is no limit to the number of attributes in one record but the size of the record cannot be higher than 400KB. The size of the record includes the attribute name and value.

Item & Item Collection

It is a record with a collection of attributes. It is like a row of SQL database table, but the structure is more flexible because you can have items with different attributes in the same table rather than rows with the same columns in the SQL table. The maximum item size in DynamoDB is 400 KB, which includes both attribute name and attribute value lengths both with binary length.

All items with the same partition key are part of an item collection. To get items collection is necessary to use the composite primary key and the sort key should be unique. No item collection can exceed 10 GB, so it's possible to run out of space for a particular partition key value.

In the image we have Customer and Order items, both share the partition key so the Item Collection has both entities.

Partition

DynamoDB stores data in partitions, each partition is like a node in a cluster. DynamoDB uses the partition key value and hash function to choose the partition to write and read the data. A partition is an allocation of storage for a table, backed by solid state drives (SSDs) and automatically replicated across multiple Availability Zones within an AWS Region. Partition management is handled entirely by DynamoDB, you never have to manage partitions yourself. A partition can store up to 10 GB of data.

Partition Key

It is an attribute and is used by DynamoDB to store the data in a specific partition and distribute write and read operations load between the partitions. DynamoDB uses a hash function with a partition key value to select the partition. It is a mandatory attribute to create a DynamoDB table and execute write and query operations over the DynamoDB table.

A good practice is using a partition key with many possible values, for example, the personal identification number is a good choice because each person has his own identification and there are many persons with identifications. Another good practice is to use PK as the partition key attribute name because it is general to all entities going to store in a single DynamoDB table.

Sort Key

It is an attribute and is used by DynamoDB to order the data inside a partition. It is an optional field, you can create a DynamoDB table without a sort key but is a good practice to use it to sort the data. Uses SK as a sort key name because it is general to all entities going to store in a single DynamoDB table.

DynamoDB uses UTF-8 character encoding to order data, If the data type of the sort key is Number, the results are returned in numeric order, otherwise, the results are returned in order of UTF-8 bytes. By default, the sort order is ascending. To reverse the order, set the ScanIndexForward parameter to false.

Local Secondary Index (LSI)

It is a table extension of the base table that contains a copy of some or all of the attributes from its base table. You can do read operations over the local secondary index, and the write operation only over the base table because DynamoDB projects the items from the base table to the local secondary index.

You can create up 5 local secondary index per DynamoDB table and only can set up these when you are creating the base table with a composite primary key (partition key and sort key). A local secondary index maintains a partition key from the base table and an alternate sort key, this is helpful when you need to sort the items with more dimensions, for example, sort the CUSTOMER by Name, created date, score, and more.

In this example, the Order item has a timestamp attribute that is the Sort Key of the Local Secondary Index, Dynamo replicates the information from Base Table to Local Secondary Index, the Partition Key is the same, Sort Key is the timestamp and the SK attribute is the Sort Key of the Base Table.

Global Secondary Index (GSI)

It is similar to Local Secondary Index, you can create it when you create the base table or after it and set up Partition Key and Sort Key. You can create up to 20 Global Secondary Index and only execute read operations, the write operation only over the base table because DynamoDB projects the items from the base table to the global secondary index when the item is created with the attribute declared as partition key and sort key(optional) of the index.

In this example, the Base Table has GSI1PK and GSI1SK attributes for the Partition Key and Sort Key of the Global Secondary Index, DynamoDB replicates the information from Base Table to Index which is the Order record, the PK and SK of the Base Table are attributes for the Index.

In this post, we cover the main concepts of DynamoDB, Table, Primary Key, Attributes, Partition, and Index, in the next post DynamoDB Basics Part 2, we are going to see the Read and Write operations, Streams, Global Table, and more about DynamoDB.

References

DynamoDB Developer Guide https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Introduction.html
DynamoDB Actions https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Operations_Amazon_DynamoDB.html

AWS Bastion Host / Jump Box

Jose Luis — Mon, 23 May 2022 03:13:07 +0000

Como administradores de sistemas debemos mantener la infraestructura actualizada aplicando parches de seguridad, instalando versiones nuevas del sistema operativo y configurando aplicaciones correctamente para fortalecer la seguridad de los recursos tecnológicos en la nube y cumplir con el modelo de responsabilidades compartidas de aws. Para realizar estas actividades en instancias de EC2 y bases de datos en RDS en necesario acceder a estos recursos de forma segura.

En este post vamos a ver como establecer conexión con instancias EC2 y bases de datos RDS que se encuentran en subredes privadas sin acceso desde Internet a través de un Bastion Host o Jump Box usando las herramientas OpenSSH y MySQL Workbench. Utilizaremos la red privada virtual (VPC) y Subredes generadas en el post Redes en AWS desde 0 para proteger la infraestructura creando entornos privados y seguros con topologías de red y controlando el acceso de comunicaciones entrantes o salientes hacia instancias de EC2 o bases de datos en RDS.

Iniciamos creando la instancia EC2 para el Bastion Host, posterior aprovisionando las instancias EC2 y RDS en Subred privada y estableciendo conexión por medio del Bastion Host a las instancias privadas de EC2 y MySQL RDS, comencemos:

Crear Bastion Host.

Es una instancia de EC2 desplegada en una VPC y Subred publica (con acceso desde internet) para la conexión con instancias EC2 o bases de datos RDS localizadas en VPC y Subredes privadas (sin acceso desde internet) por medio del protocolo SSH (Secure Shell) y TCP/IP. Los pasos para crear un Bastion Host son:

Usar asistente de lanzamiento de instancia EC2. Seleccionar AMI (Amazon Machine Image), para este ejemplo usamos Amazon Linux 2 AMI.

Seleccionar VPC y Subred Publica en configuración de detalles de la instancia EC2.

En el Post Redes en AWS desde 0 creamos la subred publicSubnet1A con CIDR 192.168.0.0/27 y VPC 192.168.0.0/24, la tabla de enrutamiento y la lista de control de acceso de red (Network ACL) permiten comunicaciones internas y externas.

Crear Security Group (Firewall) con regla de entrada para habilitar conexión SSH por medio del protocolo TCP puerto 22 y desde la dirección IP de nuestro equipo local.

Crear llaves (publica y privada) para la autenticación por conexión con SSH.

Revisar detalles de lanzamiento de la instancia y confirmar.

Crear instancia EC2 en subred privada.

Una instancia EC2 es privada si es desplegada en una subred con restricciones en la comunicación desde internet. Los pasos para crear una instancia EC2 y desplegarla en subred privada son:

Usar asistente de lanzamiento de instancia EC2. Seleccionar AMI (Amazon Machine Image), para este ejemplo usamos Amazon Linux 2 AMI.

Seleccionar VPC y Subred Privada en configuración de detalles de la instancia EC2.

En el Post Redes en AWS desde 0 creamos la subred privateSubnet1A con CIDR 192.168.0.32/27 privada y la VPC 192.168.0.0/24, la tabla de enrutamiento y la lista de control de acceso de red (Network ACL) permiten comunicaciones internas.

Crear Security Group (Firewall) con regla de entrada para habilitar conexión SSH por medio del protocolo TCP puerto 22 y desde el Security Group usado por el Bastion Host.

Crear llaves (pública y privada) para la autenticación por conexión con SSH. Estas llaves deben ser diferente a las creadas para la instancia del Bastion Host para incrementar la seguridad y mitigar el riesgo de acceso a varios recursos de infraestructura si un atacante secuestra las llaves.

Revisar detalles de lanzamiento de la instancia y confirmar.

Crear Base de Datos RDS en subred privada.

Una instancia de base de datos RDS es privada si es desplegada en una subred con restricciones en la comunicación desde internet. Los pasos para crear una base de datos SQL privada en RDS son:

Crear Subnet Group con VPC y al menos dos Subredes en 2 zonas de disponibilidad diferentes en la región AWS:

En el Post Redes en AWS desde 0 creamos la subred privada privateSubnet1A con CIDR 192.168.0.32/27 en zona de disponibilidad us-east-2a y la VPC con CIDR 192.168.0.0/24, la tabla de enrutamiento y la lista de control de acceso de red (Network ACL) permiten comunicaciones internas. En la zona de disponibilidad us-east-2b tenemos la subred publicSubnet1B con CIDR 192.168.0.64/27 en la misma VPC. El objetivo de Subnet Group es agrupar subredes que son usadas para el despliegue de instancias de la base de datos por ejemplo read replica o standby.

Crear Security Group (Firewall) con regla de entrada para habilitar conexión MYSQL/Aurora por medio del protocolo TCP puerto 3306 y desde el Security Group usado por el Bastion Host

Usar asistente de lanzamiento de instancia RDS. Seleccionar Motor de Base de Datos, para este ejemplo usamos MySQL:

Ingresar identificador de instancia de base de datos y crear credenciales de acceso a la base de datos, Master username y Master password:

Configurar propiedades de conexión a la base de datos, ingresar valores para VPC y Subnet Group (Creado en el punto 1), habilitar restricción acceso público, security group (Creado en el punto 2), seleccionar zona de disponibilidad donde se encuentra la subred privada (us-east-2) y puerto.

Asignar nombre a la base de datos:

Crear base de datos.

Conexión SSH hacia el Bastion Host

Para la conexión hacia el Bastion Host desde nuestro equipo Local podemos utilizar clientes SSH como OpenSSH, PuTTY, MobaXterm, WinSCP y otros. Necesitamos la dirección ip publica o DNS de la instancia EC2 (Bastion Host) y el nombre de usuario por defecto en la AMI usada para crear la instancia EC2. Los pasos para establecer la conexión son:

Buscar dirección IP publica o DNS en detalles de la instancia EC2:

Buscar nombre de usuario por defecto usado para crear instancia EC2. Para Amazon Linux 2 AMI el nombre de usuario es ec2-user.
Desde nuestro equipo local usando el cliente OpenSSH por línea de comandos ejecutamos el comando ssh-add -k para agregar la llave privada (.pem) del Bastion Host al agente de ssh:

En Windows, iniciar el Servicio OpenSSH Authentication Agent primero para ejecutar el comando ssh-add.

Ejecutar el comando ssh {user-name}@{public IP address}, con el nombre de usuario y la dirección ip publica de la instancia EC2 para establecer conexión ssh:

Conexión SSH hacia instancia EC2 Privada

Con la conexión establecida con el Bastion Host, ejecutamos los siguientes pasos desde el cliente OpenSSH para establecer conexión con la instancia EC2 privada::

Buscar dirección IP privada en detalles de la instancia EC2:

Nombre de Usuario por defecto de la AMI (Amazon Machine Image) utilizada en la creación de la instancia EC2. Para Amazon Linux 2 AMI el nombre de usuario es ec2-user.
Desde nuestro equipo local usando el cliente OpenSSH por línea de comandos ejecutamos el comando ssh-add -k para agregar la llave privada (.pem) de la instancia privada al agente de ssh:

En Windows, ejecutar el Servicio OpenSSH Authentication Agent primero para ejecutar el comando ssh-add.

Ejecutar el comando ssh – J {user-name}@{public IP address} {user-name}@{private IP address}, con el nombre de usuario y la dirección ip publica del Bastion Host y el nombre de usuario y la dirección ip de la instancia privada para establecer conexión ssh:

Conexión Base de datos Privada

Los pasos para establecer conexión con la base de datos privada por medio del cliente MySQL Workbench son los siguientes:

Buscar end-point y puerto de conexión a la base de datos en detalles de la instancia en RDS.

Abrir asistente para configurar conexión estándar TCP/IP sobre SSH en MySQL Workbench. Ingresar información en los siguientes parámetros:

a. SSH Hostname: Dirección ip publica o DNS del Bastion Host con el puerto 22.

b. SSH Username: Nombre de Usuario por defecto de la AMI (Amazon Machine Image) utilizada en la creación de la instancia EC2. Para Amazon Linux 2 AMI el nombre de usuario es ec2-user.

c. SSH Key File: Llave privada para la autenticación por conexión con SSH con el Bastion Host.

d. MySQL Hostname: End-point de conexión a la base de datos del punto 1.

e. Username: Nombre de usuario de la base de datos.

f. Password: Contraseña para la autenticación con la base de datos.

Probar la conexión

Conclusión

En este post vimos los pasos para establecer conexión con instancias de EC2 y RDS privadas de forma segura por medio del Bastion Host como servidor intermedio. Usamos las herramientas OpenSSH y Workbench como clientes en la conexión con la instancia de EC2 y base de datos en RDS.

References

How can I use an SSH tunnel and MySQL Workbench to connect to a private Amazon RDS MySQL DB instance that uses a public EC2 instance?

SSH to remote hosts though a proxy or bastion with ProxyJump

AWS Bastion Host / Jump Box

Jose Luis — Mon, 07 Feb 2022 02:45:13 +0000

As systems administrators, we should keep updating the infrastructure applying security patches, installing the new versions of the operation system, and setting up applications correctly to strengthen the security of cloud technologic resources and achieve AWS Shared Responsibility Model. For doing these activities on EC2 instances and RDS databases are necessary to access securely.

In this post, we are going to see how to establish the connection with EC2 instances and RDS databases that are on private subnetworks (without access from the internet) through Bastion Host or Jump Box and using OpenSSH and MySQL Workbench tools. We will use the virtual private cloud (VPC) and subnetworks created on the post AWS Networking from scratch to protect the infrastructure by creating private and safe environments with network topologies and access control inbound and outbound communications to EC2 or databases RDS.

We start to create an EC2 Bastion Host instance, then provision EC2 and RDS instances on a private subnetwork and establish a connection to these instances through Bastion Hots. Let’s start:

Create Bastion Host.

It is an EC2 instance deployed on VPC and public subnet (with access from the internet) for the connection with EC2 instance and RDS databases on VPC and private subnet (without access from the internet) through SSH (Secure Shell) and TCP/IP communication protocols. The steps to create a Bastion Host are:

Use EC2 instance wizard. Choose AMI (Amazon Machine Image), for this example we use Amazon Linux 2 AMI.

Choose VPC and public subnet on EC2 instance detail configuration.

In the Post AWS Network from scratch we created the publicSubnet1A subnet with CIDR 192.168.0.0/27 and VPC 192.168.0.0/24, the route table and network access control list (NACL) allows internal and external communications.

Create a Security Group (Firewall) with the inbound rule to enable SSH connection through TCP/IP protocol and port 22 from our local machine IP address.

Create keys (public and private) for SSH authentication.

Review instance launch details and launch it.

Create an EC2 instance on a private Subnet.

An EC2 instance is private when it is deployed on a subnet with communication restrictions from the internet. The steps to create an EC2 instance and deploy it on the private subnet are:

Use EC2 instance wizard. Choose AMI (Amazon Machine Image), for this example we use Amazon Linux 2 AMI.

Choose VPC and private subnet on EC2 instance detail configuration.

In the Post AWS Network from scratch, we created the privateSubnet1A subnet with CIDR 192.168.0.32/27 and VPC 192.168.0.0/24, the route table, and network access control list (NACL) allows only internal communications.

Create a Security Group (Firewall) with the inbound rule to enable SSH connection through TCP/IP protocol and port 22 from the security group of Bastion Host.

Create keys (public and private) for SSH authentication. These keys should be different from the Bastions Host keys to increase the security and reduce the risk to access many infrastructure resources when a third party gets access to the key.

Review instance launch details and launch it.

Create an RDS Database on a private Subnet.

An RDS Database instance is private when it is deployed on a private subnet with communication restrictions from the internet. The steps to create an RDS database and deploy it on the private subnet are:

Create a Subnet Group with VPC and at least two Subnetworks in two different availability zones in the AWS region:

In the Post AWS Network from scratch, we created the privateSubnet1A subnet with CIDR 192.168.0.32/27 on the availability zone us-east-2a and VPC 192.168.0.0/24, the route table, and network access control list (NACL) allows only internal communications. On the availability zone us-east-2b we have the subnet publicSubnet1B with CIDR 192.168.0.64/27 in the same VPC.

Create a Security Group (Firewall) with the inbound rule to enable MYSQL/Aurora connection through TCP/IP protocol and port 3306 from the security group of Bastion Host.

Use the RDS database, instance wizard. Choose the database engine, for this example we use MySQL:

Enter database instance identity and create database access credentials, Master username, and Master password:

Set up database connection properties, enter the values to VPC, Subnet Group (Created in step 1), enable public restrictions, security group (Created in step 2), choose availability zone where the private subnet is present (us-east-2) and port:

Enter database name:

Create a database.

SSH Connection to Bastion Host

For SSH connection to Bastion Host from our local machine, we can use SSH clients like OpenSSH, Putty, MobaXterm, WinSCP, and others. We need the public IP address or DNS of the EC2 instance (Bastion Host), the default user name of AMI used to create the EC2 instance, and a private SSH key. The steps to establish the connection are:

Find the public IP address or DNS in the EC2 instance details section:

Search the default user name of the AMI used to create the EC2 instance. The default user name of the Amazon Linux AMI is ec2-user.
From our local machine and using the OpenSSH command line, execute the command ssh-add -k to add Bastion Host private key (.pem) to ssh-agent:

On Windows, run the OpenSSH Authentication Agent service first to execute the command ssh-add.

Execute the command ssh {user-name}@{public IP address} with the user name and public IP address of EC2 instance to establish ssh connection:

SSH connection to EC2 private instance

With Bastion Host connection ready, we can execute the below steps from our local machine and with OpenSSH client to connect with EC2 private instance:

Find the private IP address on the EC2 instance details section:

Search the default user name of the AMI used to create the EC2 instance. The default user name of the Amazon Linux AMI is ec2-user.
From our local machine and using the OpenSSH command line, execute the command ssh-add -k to add EC2 private instance private key (.pem) to ssh-agent:

On Windows, run the OpenSSH Authentication Agent service first to execute the command ssh-add.

Execute the command ssh – J {user-name}@{public IP address} {user-name}@{private IP address}, with the Bastion Host user name and public IP address, and EC2 private instance user name and local IP address to establish SSH connection:

RDS private database connection

The steps to establish a connection with a private database using MySQL Workbench client from our local machine are:

Find the end-point and connection port of the database on RDS instance details.

Open MySQL Workbench wizard to set up TCP/IP connection over SSH. Enter the following fields:

a. SSH Hostname: Bastion Host public IP address or DNS with port 22.

b. SSH Username: The default user name of the AMI used to create the EC2 instance. The default user name of the Amazon Linux AMI is ec2-user.

c. SSH Key File: Bastion Host Private key (.pem) for SSH authentication.

d. MySQL Hostname: End-point database connection (point 1).

e. Username: Database username.

f. Password: Database password.

Test connection

Conclusion

In this post, we learned the steps to establish a connection with EC2 and RDS private instances through Bastion Host. We used the OpenSSH and Workbench tools to connect with the EC2 instance and RDS database.

References

How can I use an SSH tunnel and MySQL Workbench to connect to a private Amazon RDS MySQL DB instance that uses a public EC2 instance?

SSH to remote hosts though a proxy or bastion with ProxyJump

AWS Networking from scratch

Jose Luis — Tue, 12 Oct 2021 03:58:41 +0000

In this post, we are going to see the AWS networking foundations, networks, subnetworks, firewall, Internet Gateway, Access Control List, NAT, and other services.

Building networks, segregating the network, configuring routing, and network permissions are architectural decisions important for communications between the systems. Compute Services (EC2, ECS, EKS), Database (RDS, Aurora, ElastiCache), and others allow us to choose the network, subnetwork, availability zone, and firewall for deployment and execution of the AWS service. It is our responsibility to set up the communications needed for the systems and protect them against security attacks that impact the availability and integrity

We start from inwards to towards with the services that are base to other services. Each service has a description, scope, limits, and an example to know their capabilities and boundaries. Let's start.

Virtual Private Cloud (VPC)

Type: Networking.

Scope: Create a virtual private network with an IP addresses block (CIDR). It allows logical isolate the resources deployed in the virtual network from other services that are in the AWS cloud.

Limits: 5 VPC per AWS region. You can adjust the limit by requesting it from AWS support.

Example: To create a VPC you need an IPv4 or IPv6 CIDR block, this value depends on the number of subnetworks and hosts you need. We use CIDR block 192.168.0.0/24 to create VPC on the Ohio AWS region. When you create a VPC by default create Route Table and Network Access Control List (NACL).

Subnets

Type: Networking.

Scope: Create virtual subnetwork inside VPC network to split the network and deploy systems in the subnetwork. You can choose the availability zone (AZ) of the AWS region to create the subnetwork and you can create more than one subnetwork in the same AZ.

Limits: 200 subnets per VPC. You can adjust the limit by requesting it from AWS support.

Example: To create a VPC you need an IPv4 CIDR block that is inside the IPv4 CIDR block of VPC.

Taken 3 bits of VPC network address then we can create 8 subnets (2^3=8). The IPv4 CIDR block for each subnet are:

192.168.0.0/27
192.168.0.32/27
192.168.0.64/27
192.168.0.96/27
192.168.0.128/27
192.168.0.160/27
192.168.0.192/27

The other 5 bits are to create hosts, the number of hosts per subnet is 27 (2^5 = 32 - 5). AWS reserves 5 IP addresses in each subnet, the first 4 addresses and the last one.

For the 192.168.0.0/27 CIDR block the IP addresses reserved by AWS are:

192.168.0.0, Subnetwork address.
192.168.0.1, reserved by AWS for VPC Router.
192.168.0.2, reserved by AWS for Amazon-provided DNS.
192.168.0.3, reserved by AWS for future use.
192.168.0.31, Broadcast, AWS does not support Broadcast communication inside VPC.

For the 192.168.0.32/27 CIDR block the IP addresses reserved by AWS are:

192.168.0.32, Subnetwork address.
192.168.0.33, reserved by AWS for VPC Router.
192.168.0.34, reserved by AWS for Amazon-provided DNS.
192.168.0.35, reserved by AWS for future use.
192.168.0.63, Broadcast, AWS does not support Broadcast communication inside VPC.

We are going to use the 192.168.0.0/27 CIDR block for Public Subnet (Internet access) and 192.168.0.32/27 CIDR block for Private Subnet (Internal access) on availability zone us-east-2a.

Internet Gateway

Type: Networking.

Scope: Internet access gateway for the resources deployed in subnet and VPC. Also, it is a NAT for instances (EC2, RDS) that has a public IP address.

Limits: 5 Internet Gateway per AWS Region. You can adjust the limit by requesting it from AWS support.

Example: Create Internet Gateway for the VPC 192.168.0.0/24. Just can attach one Internet Gateway per VPC.

Route Table

Type: Networking.

Scope: Create routing rules to address the communications from the resource inside subnet and VPC to destination. Route Table attaches to VPC applies to all subnets of the VPC. Route Table attach directly to subnet has priority over Route Table of VPC.

Limits: 200 Route Tables per VPC and 50 routing rules by Route Table. You can adjust the limit by requesting it from AWS support.

Example: Create and attach new Route Table on Public Subnet (192.168.0.0/27) with routing rules to Internet and VPC network destinations through Internet Gateway and local routing:

For Private Subnet (192.168.0.32/27) create a new Route Table and attach to it with routing rule to VPC network destination through local routing:

Network Access Control List (NACL)

Type: Security.

Scope: Network firewall to control inbound and outbound communication to resources deployed in the subnet using rules. The rule specifies communication protocol, port, source/destination, and allow or deny communication. The rules are sorted by number, have precedence, and are stateless, you need to create inbound and outbound rules to requests and responses.

Limits: 200 NACL per VPC and 20 rules to IPv4 or IPV6 per NACL. You can adjust the limit by requesting it from AWS support.

Example: For Public Subnet (192.168.0.0/27) create and attach NACL with inbound and outbound rules to allow internet access (All traffic).

For Private Subnet (192.168.0.32/27) create and attach NACL with inbound and outbound rules to allow communications from and to Public Subnet (192.168.0.0/27).

Security Group

Type: Security.

Scope: Instance firewall to control inbound and outbound communication to instances (EC2, RDS) using rules. The rule specifies communication protocol, port, and source/destination. The rules are stateful you just need to create inbound/outbound rules for requests received/generated by the instance and is not necessary to create rules for the response.

Limits: 2500 Security Groups per AWS Region and 60 inbound and outbound rules per Security Group. You can adjust the limit by requesting it from AWS support.

Example: For EC2 instance access over internet using HTTP and SSH protocols, deploy on Public Subnet (192.168.0.0/27), create and attach Security Group with inbound rules with sources anywhere and ip address (your ip).

For EC2 instance to internal access only (no internet access), deploy on Private Subnet (192.168.0.32/27), create and attach Security Group with an inbound rule that allows communication from Security Group EC2 Instance of Public Subnet (192.168.0.0/27) over TCP protocol and port 22.

For RDS instance to internal access only (no Internet access), deploy on Private Subnet (192.168.0.32/27), create and attach Security Group with an inbound rule that allows communication from Security Group EC2 Instance of Public Subnet (192.168.0.0/27) and Security Group EC2 Instance of Private Subnet (192.168.0.32/27), both of them over TCP protocol and port 3306.

The EC2 instance on Public Subnet is a Bastion Host (Bridge) to connect to EC2 and RDS instances on Private Subnet witout internet access.

NAT Instance & NAT Gateway

Type: Networking

Scope: Internet access from instances on Private Subnet (without Internet Access) through exchange of ip addresses. Taken the ip address of communication packages send by instances and replace it with the ip address of NAT.

NAT Instance: It is EC2 instances with preconfigured Amazon Machine Image (AMI) and elastic ip address or public ip. The bandwidth depends on the bandwidth of the instance type.

NAT Gateway: Is a NAT service managed by AWS with high availability and elastic ip. Scale up to 45 Gbps in bandwidth. It can handle communication to internet, between VPCs and between VPC and on-premise corporate network.

Limits NAT Instance: Apply EC2 limits.

Limits NAT Gateway: 5 NAT Gateway per availability zone of AWS region. You can adjust the limit by requesting it to AWS support.

Example:

NAT Instance: Create EC2 instance with NAT AMI and deploy on Public Subnet (192.168.0.0/27) with internet access through Internet Gateway.

On Route Table of Private Subnet (192.168.0.32/27) add routing rule to NAT instance when destination is internet.

NAT Gateway: Create NAT Gateway and deploy on Public Subnet (192.168.0.0/27) with internet access through Internet Gateway.

On Route Table of Private Subnet (192.168.0.32/27) add routing rule to NAT Gateway when destination is internet.

Update NACL of Private Subnet (192.168.0.32/27) with inbound and outbound rules that allow internet communications (Request and Response).

Route Table and NACL of Public Subnet (192.168.0.0/27) without changes.

Conclusion

In this post we saw the main networking and security aws services to build a network and allow or deny the communication between the systems deployed on it. We also build a solution to protect the instances by restricting access to and from the internet with network and instance Firewall and allowing connection through Bastion Host and NAT.

References

VPC Limites
NAT Comparison

Redes en AWS desde 0

Jose Luis — Mon, 11 Oct 2021 05:23:17 +0000

En este artículo vamos a ver los fundamentos de las comunicaciones en AWS, redes, subredes, firewall, internet gateway, listas de control de acceso, NAT entre otros.

Crear redes, segmentos de red, configurar enrutamiento y asignar permisos de red son decisiones arquitecturales importantes para la comunicación entre los sistemas. Servicio de Computo (EC2, ECS, EKS), Bases de Datos (RDS, Aurora, ElastiCache) entre otros, permiten seleccionar la red, subred, firewall y zona de disponibilidad para el despliegue y ejecución del servicio de AWS. Es nuestra responsabilidad configurar las comunicaciones necesarias para la operación de los sistemas y protegerlos ante ataques que afecten la disponibilidad e integridad.

Iniciamos de adentro hacia fuera, con los servicios que son base para la creación de otros servicios. En cada servicio se describe el tipo, alcance, límites y un ejemplo para conocer capacidades y fronteras de cada uno. Comencemos.

Virtual Private Cloud (VPC)

Tipo: Comunicaciones

Alcance: Crear red virtual privada con bloque de direcciones ip (CIDR). Permite aislar lógicamente los recursos provisionados y desplegados en la red virtual de otros servicios que están en la nube de AWS.

Limites: Por región de AWS se pueden crear máximo 5 VPC. Pude ajustar el límite por medio de una solicitud a soporte AWS.

Ejemplo: Para crear una VPC se necesita un bloque de direcciones ip privadas (CIDR) para IPv4 o IPv6, este valor depende de la cantidad de subredes y hosts que necesitamos. Utilizaremos el bloque 192.168.0.0/24 para crear VPC en la Región de Ohio en AWS. Al crear una VPC, se crean un Route Table y Network Access Control List (NACL) automáticamente

Subnets

Tipo: Comunicaciones

Alcance: Crear subred virtual en la red VPC para segmentar la red y habilitar el despliegue de sistemas en la subred. Permite seleccionar la zona de disponibilidad de la Región de AWS para crear la subred, es posible crear más de una subred en una zona de disponibilidad.

Limites: Por VPC se puede crear hasta 200 Subredes. Pude ajustar el límite por medio de una solicitud a soporte AWS.

Ejemplo: Para crear una subred se necesita un bloque de direcciones ip (CIDR) que este dentro del bloque de direcciones asignado a la VPC.

Tomando 3 bits de la dirección de red VPC 192.168.0.0/24, podemos crear 8 subredes (2^3=8). Los bloques de direcciones ip para cada subred son:

192.168.0.0/27
192.168.0.32/27
192.168.0.64/27
192.168.0.96/27
192.168.0.128/27
192.168.0.160/27
192.168.0.192/27

Los restantes 5 bites son para crear hosts, la cantidad de hosts en cada subred es 27 (2^5 = 32 - 5). En cada bloque de direcciones ip de las subredes, AWS reserva 5 direcciones ip, las 4 primeras direcciones y la última.

Para el bloque 192.168.0.0/27, las direcciones ip reservadas por AWS son:

192.168.0.0, Dirección de red.
192.168.0.1, Reservada por AWS para VPC Router.
192.168.0.2, Reservada por AWS para DNS.
192.168.0.3, Reservada por AWS para uso futuro.
192.168.0.31, Broadcast, AWS no soporta comunicación broadcast en una VPC.

Para el bloque 192.168.0.32/27, las direcciones ip reservadas por AWS son son:

192.168.0.32, Dirección de red.
192.168.0.33, Reservada por AWS para VPC Router.
192.168.0.34, Reservada por AWS para DNS.
192.168.0.35, Reservada por AWS para uso futuro.
192.168.0.63, Broadcast, AWS no soporta broadcast en una VPC.

Utilizamos el bloque CIDR 192.168.0.0/27 para crear Subred Publica (acceso a Internet) y el bloque CIDR 192.168.0.32/27 para crear Subred Privada (acceso interno) en la zona de disponibilidad us-east-2a de Ohio.

Internet Gateway

Tipo: Comunicaciones

Alcance: Puerta de entrada de acceso a Internet para los recursos desplegados en la subred y VPC. También es un NAT para las instancias (EC2, RDS) que tiene direcciones ip públicas.

Limites: Hasta 5 Internet Gateway por Región. Pude ajustar el límite por medio de una solicitud a soporte AWS.

Ejemplo: Se crea Internet Gateway para VPC 192.168.0.0/24. Un Internet Gateway se adjunta a una sola VPC.

Route Table

Tipo: Comunicaciones

Alcance: Crear reglas de enrutamiento usadas para direccionar las comunicaciones de los recursos en al VPC y Subred hacia un sistema destinatario. Route Table adjunto a una VPC aplica para todas las Subredes de la VPC. Route Table adjunto a una Subred tiene prioridad sobre Route Table de la VPC.

Limites: Hasta 200 Route Tables por VPC y máximo 50 reglas de enrutamiento por Route Table. Pude ajustar el límite por medio de una solicitud a soporte AWS.

Ejemplo: En Subred Publica 192.168.0.0/27 se crea y asocia Route Table con reglas de enrutamiento hacia Internet Gateway y red interna Local.

Para Subred Privada 192.168.0.32/27 se crea y asocia Route Table con regla de enrutamiento hacia red interna Local.

Network Access Control List (NACL)

Tipo: Seguridad

Alcance: Firewall de red para controlar comunicaciones de entrada y salida a recursos en la Subred por medio de reglas. Una regla especifica el protocolo de comunicación, puerto, origen/destino y permitir o rechazar la comunicación. Las reglas son ordenadas por número, tienen precedencia y son sin estado, se crean reglas de entrada y salida para peticiones y respuestas

Limites: Máximo 200 Network ACL por VPC y hasta 20 reglas de IPv4 o IPv6 por NACL. Pude ajustar el límite por medio de una solicitud a soporte AWS.

Ejemplo: En Subred Publica 192.168.0.0/27 se crea y asocia NACL con reglas de entrada y salida que permitan acceso a internet (todas las comunicaciones).

Para Subred Privada 192.168.0.32/27 se crea y asocia NACL con regla de entrada y salida que permitan las comunicaciones desde y hacia la Subred Publica 192.168.0.0/27.

Security Group

Tipo: Seguridad

Alcance: Firewall de instancia para controlar comunicaciones de entrada y salida de una instancia (EC2, RDS) por medio de reglas. Una regla especifica el protocolo de comunicación, puerto y origen/destino especifico. Las reglas de Security Group son con estado, solo se necesitan crear reglas de entrada/salida para peticiones recibidas/generadas por la instancia y no necesita reglas para la respuesta.

Limites: Máximo 2500 Security Group por Region de AWS y hasta 60 reglas de entrada o salida por Security Group. Pude ajustar el límite por medio de una solicitud a soporte AWS.

Ejemplo: Para instancia EC2 con conexión desde internet por medio de los protocolos de comunicación HTTP y SSH, desplegar en Subred Publica 192.168.0.0/27, crear y asociar Security Group con reglas de entrada que permitan comunicaciones por SSH y HTTP desde Internet e ip especifica.

Para Instancia EC2 con acceso interno (sin acceso desde internet), desplegar en Subred Privada 192.168.0.32/27, crear y asociar Security Group con regla de entrada que permitan comunicaciones desde Security Group de la instancia EC2 en la Subred Publica 192.168.0.0/27 por el protocolo TCP y puerto 22.

Para Instancia RDS con acceso interno (sin acceso desde internet), desplegar en Subred Privada 192.168.0.32/27, crear y asociar Security Group con reglas de entrada que permitan comunicaciones desde Security Group de la instancia EC2 en la Subred Publica 192.168.0.0/27 y el Security Group de la instancia EC2 en la Subred Privada 192.168.0.32/27, ambos por el protocolo TCP y puerto 3306.

La instancia EC2 en Subred Publica es Bastion Host (Puente) para conectar a las instancias EC2 y RDS en Subred Privada sin acceso desde internet.

NAT Instance & NAT Gateway

Tipo: Comunicaciones

Alcance: Acceso a Internet para las instancias que se encuentran en Subred privada (sin acceso a interne) por medio del cambio de direcciones ip. Toma la dirección ip de los paquetes de comunicaciones enviados por las instancias y las reemplaza con la dirección ip de la NAT.

NAT Instance: Es una instancia EC2 con imagen de Maquina Amazon (AMI) preconfigurada, se despliega en subred pública con dirección ip publica o elástica. Ancho de banda depende del ancho de banda de la instancia EC2.

NAT Gateway: Es un servicio NAT administrado por AWS, con alta disponibilidad y dirección ip elástica. Escala hasta 45GB de ancho de banda. Permite comunicaciones hacia internet, entre VPCs y entre VPC y redes corporativas.

Limites NAT Instance: Aplican los límites de instancias EC2.

Limites NAT Gateway: Hasta 5 NAT Gateway por zona de disponibilidad de la región AWS. Pude ajustar el límite por medio de una solicitud a soporte AWS.

Ejemplo:

NAT Instance: : En Subred Publica 192.168.0.0/27 desplegamos instancia EC2 con imagen NAT.

En Route Table de la Subred Privada 192.168.0.32/27, agregamos regla de enrutamiento hacía NAT instance cuando el destino es internet.

NAT Gateway: Se crea NAT Gateway y se despliega en Subred Publica 192.168.0.0/27.

En Route Table de la Subred Privada 192.168.0.32/27, agregamos regla de enrutamiento hacía NAT instance cuando el destino es internet.

Actualizar NACL de Subred Privada 192.168.0.32/27 con regla de entrada y salida que permitan las comunicaciones hacia Internet (Peticiones y Respuestas).

Route Table y NACL de Subred Publica 192.168.0.0/27 sin cambios.

Conclusiones

En este post vimos los principales servicios de redes y seguridad de aws para construir redes y controlar las comunicaciones entre los sistemas desplegados en las red. También construimos solución para proteger las instancias restringiendo el acceso desde y hacia internet con Firewall de red e instancia y permitiendo la conexión por medio de Bastion Host y NAT.

Referencias

VPC Limites
Comparación NAT

DynamoDB Basics, Part 2

Jose Luis — Tue, 04 May 2021 15:06:49 +0000

Continuing with the post on DynamoDB Basics, Part 1, we are going to see the operations you can execute over the DynamoDB table and the features you can use the improve the performance and availability, let's start.

Read operations

There are two operations, Query and Scan to perform over the main table or secondary Indexes, the detail of each operation are:

Query: You need the partition key to choose the partition and execute the operation over it. You can use the sort key and add filters to get specific data. It reads multiple items that have the same partition key and satisfies the sort key constraint when it is present. The filters are constraints over the attributes and these are validated after the query operation.

A single Query operation will read up 1 MB of data, you can set the maximum number of items also. When the query result size is more than 1 MB the Query results are divided into pages of data that are 1 MB in size or less.

Scan: You don’t need the partition key to perform it, you can add filters to define constraints over the attributes including the primary key (PK + SK) but these are validated after the scan operations which means it is an expensive operation because read all items in each partition and then apply the filters. When the total number of scanned items exceeds the maximum dataset size limit of 1 MB, the scan stops and the results are returned to the user as a LastEvaluatedKey value to continue the scan in a subsequent operation.

Write operations

Create, Update, and Delete are operations you can execute ONLY over the DynamoDB main table. If the table has a secondary index, then DynamoDB performs the write operation over the index. When you create an item with an existing primary key DynamoDB updates the item in the table, you can add a conditional to execute it only if there is not already an item with the same primary key. You cannot update the primary key of one item, you need first delete it and create with the new primary key, DynamoDB does not support updating the primary key in one shot.

Provisioned Capacity

You can set up on-demand provisioned capacity to pay just for the write and query operations you perform or provisioned to specify the number of data reads and writes per second that you require for your application. On-demand is more expensive than provisioned, select provisioned to save on throughput costs if you can reliably estimate your application's throughput requirements. Both are using the Write Capacity Units (WCU) and Read Capacity Unit (RCU) to execute operations over DynamoDB main table and index.

Read Capacity Unit (RCU)

All the read operations over the DynamoDB main table and secondary indexes spend read capacity units, this a read per second. The RCU is shared between the main table and local secondary index, for the global secondary index you can set up RCU. One RCU represents one strongly consistent read request per second, or two eventually consistent read requests per second, for an item up to 4 KB in size. Two RCU represent one transactional read for items up to 4 KB. If you need to read an item that is larger than 4 KB, DynamoDB needs additional read request units.

The results returned by the Query operation could be item collection or single item depending on the query expression used, in both case the number of RCU depends on the size of the result not the number of items, for example, query operation using the partition key return 4 items (all with the same partition key), each item size is 1 KB, total size is 4 KB for strong consistent read you need 1 RCU or half for eventually consistent read requests

Write Capacity Unit (WCU)

All the write operations are over the DynamoDB main table only, DynamoDB executes write operations over secondary indexes automatically. One WCU represents one write per second for an item up to 1 KB in size. If you need to write an item that is larger than 1 KB, DynamoDB must consume additional write capacity units. The total number of write capacity units required depends on the item size not the number of items impacted. For example, if your item size is 2 KB, you require 2 write capacity units to sustain one write request per second. DynamoDB spends more WCU when need to project the item attributes from the main table to the secondary index, you will need more WCU.

Eventually Consistent Reads

When you read data from a DynamoDB table, the response might not reflect the results of a recently completed write operation. The response might include some stale data. If you repeat your read request after a short time, the response should return the latest data. By default, the read operations use eventually consistent read. The read operation with eventually consistently spends less RCU, you can perform two reads per second for items up to 4KB.

Strongly Consistent Reads

When you request a strongly consistent read, DynamoDB returns a response with the most up-to-date data, reflecting the updates from all prior write successful operations. Strongly consistent reads use more throughput capacity than eventually consistent reads, you can perform 1 request per second for items up to 4KB, and may have higher latency than eventually consistent reads.

Transactional

You can group multiple actions together and submit them as a single all-or-nothing TransactWriteItems or TransactGetItems operation.

TransactWriteItems is a synchronous and idempotent write operation that groups up to 25 write actions in a single all-or-nothing operation. These actions can target up to 25 distinct items in one or more DynamoDB tables within the same AWS account and in the same Region. The aggregate size of the items in the transaction cannot exceed 4 MB.

TransactGetItems is a synchronous read operation that groups up to 25 Get actions together. These actions can target up to 25 distinct items in one or more DynamoDB tables within the same AWS account and Region. The aggregate size of the items in the transaction can't exceed 4 MB.

Global Table

DynamoDB replicates the data between the availability zone of the aws region, to replicate the data between the aws region you need Global Table. DynamoDB replicates write operations from the main table and secondary index from one region to another region to improve the availability of your data. Replicate data between regions and also spend Write Capacity Unit.

Stream

All the write operations over DynamoDB table generate a stream with detail of the old and new items, or only old or new items, you can choose. Behind AWS DynamoDB table there is a Kinesis Data Stream managed by AWS where all the stream is going on, so you can integrate the lambda function with the source event stream to read the stream.

The above elements are the most important, and most of them are tools used to DynamoDB design data models and build client applications to perform read and write operations, hope this post can help you to understand DynamoDB.

References

DynamoDB Developer Guide https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Introduction.html
DynamoDB Actions https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Operations_Amazon_DynamoDB.html
Alex DeBrie Post The What, Why, and When of Single-Table Design with DynamoDB https://www.alexdebrie.com/posts/dynamodb-single-table/