Forem: Salaudeen O. Abdulrasaq

AWS Security Groups Now Show Related Resources

Salaudeen O. Abdulrasaq — Thu, 12 Feb 2026 11:06:52 +0000

AWS has quietly rolled out a useful update to the EC2 Security Groups console, you can now see which resources are associated with a security group directly from the SG details page.

What's New?

There's a new "Related resources" tab that scans your AWS resources and displays which ones are using that specific security group. In the example above, it found 2 resources (an ENI and an EC2 instance) linked to this jump-box-sg security group out of 69 total resources scanned.

Why This Matters

If you've ever tried to delete or modify a security group and wondered "what's actually using this?" this feature is for you. Previously, you'd need to:

Run CLI commands with filters
Search through multiple services manually
Use third-party tools or scripts

Now it's one click away.

Use Cases

Auditing: Quickly identify orphaned or over-permissive security groups

Impact analysis: Know exactly what you'll affect before modifying rules

Cleanup: Confidently delete unused security groups

Compliance: Document which resources share network access policies

Small quality-of-life improvements like this make a real difference in day-to-day infrastructure management.

Building Highly Available Vault on a Budget: Raft + MinIO for Resilience

Salaudeen O. Abdulrasaq — Sun, 05 Oct 2025 07:58:12 +0000

Event: HashiTalks: Africa 2025

Category: DevOps, HashiCorp Vault, Automation

Modern infrastructure depends on secure secret management, yet achieving high availability (HA) often feels like a luxury reserved for enterprise budgets.

In this blog, we’ll explore how you can build a highly available, production-grade HashiCorp Vault cluster using Raft Integrated Storage and the community edition of MinIO (an S3-compatible object storage) for resilient backups. Everything was implemented in my homelab.

Why This Approach?

Many organizations, especially those operating on tight budgets or in hybrid environments, require strong security but cannot always afford enterprise secret management service.

Instead of depending on Vault Enterprise with external storage solutions such as Consul, etcd, or DynamoDB, my approach leverages:

Vault OSS (Open Source) for secret management
Raft storage for native leader election and data replication
MinIO for S3-compatible snapshot backups

This setup provides HA, fault tolerance, and easy disaster recovery, all without extra licensing costs.

Architecture Overview

Here’s what the setup looks like:

Each Vault node runs in HA mode using Raft. The leader handles writing and replicating data to followers automatically.
Snapshots are taken periodically and pushed to MinIO for offsite/off-cluster recovery.

Note: All three servers used in this setup are provisioned within a homelab environment.

Step 1: Prepare Your Environment

Provision 3 servers (VMs or bare metal).
Ensure network connectivity between them and install:

sudo apt install vault
sudo mkdir -p /opt/vault/tls /opt/vault/data /opt/vault/backups

Create a Local Certificate Authority (CA)
Vault requires TLS for secure communication. I’ll start by generating my own self-signed CA and use it to issue Vault’s server certificate.

Generate a Private Key for the CA

openssl genrsa -out vault-ca-key.pem 4096

Create a Self-Signed CA Certificate

openssl req -x509 -new -nodes -key vault-ca-key.pem \
  -subj "/CN=Vault-CA" \
  -days 3650 -out vault-ca-cert.pem

Generate a Vault Server Private Key and CSR; This should be done for all the vault servers.

openssl genrsa -out vault-server-key.pem 2048

openssl req -new -key vault-server1.key -out vault-server-1.csr -config vault-server1.conf

Below is the content of vault-server1.conf; update accordingly for other vault instances.

~/vault-cert/vault-ca (0.146s)
cat vault-server1.conf
[req]
default_bits = 2048
prompt = no
default_md = sha256
req_extensions = v3_req
distinguished_name = dn

[dn]
C = NG
ST = Abuja
L = Abuja
O = VaultOrg
OU = DevOps
CN = vault-server-1

[v3_req]
subjectAltName = @alt_names

[alt_names]
DNS.1 = vault-server-1
DNS.2 = vault1.internal.local
DNS.3 = 192-168-64-5.nip.io
IP.1 = 192.168.64.5

Sign the Server Certificate with the CA

openssl x509 -req -in vault-server-1.csr \
  -CA vault-ca-cert.pem -CAkey vault-ca-key.pem -CAcreateserial \
  -out vault-server-cert1.pem -days 365 -sha256

Copy Certificates

Move your generated certs to /opt/vault/tls:

sudo mkdir -p /opt/vault/tls
sudo cp vault-server-cert1.pem vault-server-key.pem vault-ca-cert.pem /opt/vault/tls

Step 2: Configure Vault for HA Using Raft Storage

Configure accordingly on all three vault servers.
make the certificate name on each server uniform as shown in the vault.hcl file below.
Edit /etc/vault.d/vault.hcl:

root@vault-server-1:/opt/vault/backups# cat /etc/vault.d/vault.hcl
1 ui = true
2 disable_mlock = true
3
4 storage "raft" {
5   path = "/opt/vault/data"
6   node_id = "vault-server-1"
7 
8   retry_join {
9     leader_tls_servername = "vault-server-1"
10     leader_api_addr = "https://192.168.64.5:8200"
11     leader_ca_cert_file = "/opt/vault/tls/vault-ca.pem"
12     leader_client_cert_file = "/opt/vault/tls/vault-cert.pem"
13     leader_client_key_file = "/opt/vault/tls/vault-key.pem"
14   }
15 
16   retry_join {
17     leader_tls_servername = "vault-server-2"
18     leader_api_addr = "https://192.168.64.6:8200"
19     leader_ca_cert_file = "/opt/vault/tls/vault-ca.pem"
20     leader_client_cert_file = "/opt/vault/tls/vault-cert.pem"
21     leader_client_key_file = "/opt/vault/tls/vault-key.pem"
22   }
23 
24   retry_join {
25     leader_tls_servername = "vault-server-3"
26     leader_api_addr = "https://192.168.64.7:8200"
27     leader_ca_cert_file = "/opt/vault/tls/vault-ca.pem"
28     leader_client_cert_file = "/opt/vault/tls/vault-cert.pem"
29     leader_client_key_file = "/opt/vault/tls/vault-key.pem"
30   }
31 }
32 
33 listener "tcp" {
34   address = "0.0.0.0:8200"
35   tls_cert_file = "/opt/vault/tls/vault-cert.pem"
36   tls_key_file = "/opt/vault/tls/vault-key.pem"
37 }
38 
39 api_addr = "https://192.168.64.5:8200"
40 cluster_addr = "https://192.168.64.5:8201"

Start and enable Vault:

systemctl enable vault
systemctl start vault

root@vault-server-1:/opt/vault/tls# systemctl status vault
● vault.service - "HashiCorp Vault - A tool for managing secrets"
     Loaded: loaded (/usr/lib/systemd/system/vault.service; enabled; preset: enabled)
     Active: active (running) since Sun 2025-09-28 18:53:51 UTC; 23s ago
   Main PID: 8944 (vault)
     Tasks: 8 (limit: 4549)
    Memory: 23.9M (peak: 24.1M)
     CPU: 750ms
  CGroup: /system.slice/vault.service
          └8944 /usr/bin/vault server -config=/etc/vault.d/vault.hcl

Initialize and unseal Vault, then join other nodes to form a Raft cluster.

Initialize Vault on one of the vault nodes

$ vault operator init -key-shares=5 -key-threshold=3
Get "https://192.168.64.5:8200/v1/sys/seal-status": tls: failed to verify certificate: x509: certificate signed by unknown authority

$ export VAULT_ADDR="https://192.168.64.5:8200"
$ export VAULT_CACERT="/opt/vault/tls/vault-ca.pem"

$ echo 'export VAULT_CACERT="/opt/vault/tls/vault-ca.pem"' > ~/.bashrc
$ source ~/.bashrc

$ vault operator init -key-shares=5 -key-threshold=3
Unseal Key 1: oh0oYmK6oQkdU9oPQF35nMYKgZoZEhFHS40adcTK13Xh
Unseal Key 2: I7gby7hcwW/njhGiPVLOVBTBgUQGk50C7+FQZKtLKUvq
Unseal Key 3: OzeDrBJJ@epEonEFc69Nctj5uvhW+u9+tVMEo4jQSkeP
Unseal Key 4: ryf5ZXCmaulkseIYmOXIRAavOWEOvDdSbu7sdinUvnfU
Unseal Key 5: 87BDV1ithJhk8Abtb5SIDpeXtdH3/p70nwc650S1eUm+
Initial Root Token: hvs.zK4B84b2f5pyCgeYtv5WG8RR

Vault initialized with 5 key shares and a key threshold of 3. Please securely distribute the key shares printed above. When the Vault is re-sealed, restarted, or stopped, you must supply at least 3 of these keys to unseal it before it can start servicing requests.

Vault does not store the generated root key. Without at least 3 keys to reconstruct the root key, Vault will remain permanently sealed!

It is possible to generate new unseal keys, provided you have a quorum of existing unseal keys shares. See "vault operator rekey" for more information.

root@vault-server-1:/opt/vault/tls# vault operator unseal
Unseal Key (will be hidden): 
Key Value
--- ----
Seal Type shamir
Initialized true
Sealed true
Total Shares 5
Threshold 3
Unseal Progress 2/3
Unseal Nonce d4ab9730-3088-e2b7-33bc-b4e69ec1b568
Version 1.20.4
Build Date 2025-09-23T13:22:38Z
Storage Type raft
Removed From Cluster false
HA Enabled true

Run vault status command to view the current state of the cluster

root@vault-server-1:/opt/vault/tls# vault status
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 5
Threshold 3
Version 1.20.4
Build Date 2025-09-23T13:22:38Z
Storage Type raft
Cluster Name vault-cluster-edcc5476
Cluster ID 130c35ea-023d-c34e-4d69-c217fa974d04
Removed From Cluster false
HA Enabled true
HA Cluster https://192.168.64.5:8201
HA Mode active
Active Since 2025-09-28T19:11:36.208910265Z
Raft Committed Index 37
Raft Applied Index 37

Verify Cluster

Check Raft peers:

vault operator raft list-peers

Get current leader: vault operator raft list-peers

root@vault-server-1:/opt/vault/tls# vault operator raft list-peers
Node        Address             State  Voter
vault-server-1  192.168.64.5:8201  leader  true
vault-server-2  192.168.64.6:8201  follower  true
vault-server-3  192.168.64.7:8201  follower  true
root@vault-server-1:/opt/vault/tls#

Open the UI page of the vault and log in with root token (this token was generated alongside sealed keys during initialization) to further confirm status.

Step 3: Enable KV Secrets Engine (Version 2)

Run the below command on the leader node.

vault secrets enable -path=secret kv-v2

Create a key-value pair:

vault kv put secret/demo event="HashiTalksAfrica" year="2025" location="Online"

Verify:

vault kv get secret/demo

===== Secret Path =====
secret/data/demo

======= Metadata =======
Key                Value
---                -----
created_time       2025-10-04T07:42:19.123456Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

========== Data ==========
Key         Value
---         -----
event       HashiTalksAfrica
year        2025
location    Online

Step 4: Automate Vault Snapshot Backups to MinIO

Follow the link to install MinIO on a server (as a container or system service)or use AWS S3 if you prefer it.

root@vault-server-2:~# ssh root@192.168.64.4
The authenticity of host '192.168.64.4 (192.168.64.4)' can't be established.
ECDSA key fingerprint is SHA256:Ld5Gu7HBMnLLTYhVQzFvuYfPODcUwEZOHLBKXXXXXX.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.168.64.4' (ECDSA) to the list of known hosts.
root@192.168.64.4's password: 

root@minio-server:~# mkdir /opt/minio
root@minio-server:~# cd /opt/minio

# Download the latest MinIO binary for Linux/amd64
root@minio-server:/opt/minio# wget https://dl.min.io/server/minio/release/linux-amd64/minio
--2025-10-08 06:46:00--  https://dl.min.io/server/minio/release/linux-amd64/minio
Resolving dl.min.io (dl.min.io)... 178.128.69.202, 138.68.11.125
Connecting to dl.min.io (dl.min.io)|178.128.69.202|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 74123264 (71M) [application/octet-stream]
Saving to: 'minio'

minio                   100%[=======================] 70.71M  4.14MB/s    in 17s     

2025-10-08 06:46:17 (4.14 MB/s) - 'minio' saved [74123264/74123264]

# Make the binary executable
root@minio-server:/opt/minio# chmod +x minio

# Create a systemd service file for MinIO
root@minio-server:/opt/minio# cat > /etc/systemd/system/minio.service <<EOF
[Unit]
Description=MinIO
Documentation=https://docs.min.io
Wants=network-online.target
After=network-online.target
AssertFileIsExecutable=/opt/minio/minio

[Service]
WorkingDirectory=/opt/minio/
ExecStart=/opt/minio/minio server /opt/minio/data
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

# Start and enable the MinIO service
root@minio-server:/opt/minio# systemctl enable minio
Created symlink /etc/systemd/system/multi-user.target.wants/minio.service → /etc/systemd/system/minio.service.
root@minio-server:/opt/minio# systemctl start minio

Install MinIO client (mc) on all vault servers; This client will enable the vault server to run mc command against the minIO server.

# Download the correct ARM64 version
wget https://dl.min.io/client/mc/release/linux-arm64/mc -O /usr/local/bin/mc

# Make it executable
chmod +x /usr/local/bin/mc

# Verify installation
mc --version
-- 2025-10-05 06:40:05 -- https://dl.min.io/client/mc/release/linux-arm64/mc
Resolving dl.min.io (dl.min.io)... 178.128.69.202, 138.68.11.125
Connecting to dl.min.io (dl.min.io)|178.128.69.202|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 28704952 (27M) [application/octet-stream]
Saving to: '/usr/local/bin/mc'

/usr/local/bin/mc        100%[=======================]  27.37M  2.71MB/s    in 10s     

2025-10-05 06:40:16 (2.71 MB/s) - '/usr/local/bin/mc' saved [28704952/28704952]

mc version RELEASE.2025-08-13T08-35-41Z (commit-id=7394ce0dd2a80935aded936b09fa12cbb3cb8096)
Runtime: go1.24.6 linux/arm64
Copyright (c) 2015-2025 MinIO, Inc.
License GNU AGPLv3 <https://www.gnu.org/licenses/agpl-3.0.html>
root@vault-server-2:~#

Add MinIO endpoint alias on all vault servers

Create a minio bucket with the name vault using the below command:
PS: You can run the command on the vault node or on the MinIO server, you can also create the bucket from the UI.

# Create the bucket
root@vault-server-1:~# mc mb myminio/vault
root@vault-server-1:~# mc mb myminio/vault
[2025-28-08 20:55:07 UTC]   0B vault

Step 5: Verify Cluster Backup

Firstly, create a backup manually and push to minio before automating the process with a shell script vault-backup-to-minio.sh, systemd service and a systemd timer to trigger the service at predefined intervals.

Automating Backup

Backup Script — /usr/local/bin/vault-backup-to-minio.sh

PS: configure the below steps on all the vault instances.

#!/usr/bin/env bash
set -euo pipefail

#########################CONFIGURATION########################
VAULT_ADDR="${VAULT_ADDR:-https://192.168.64.5:8200}"
VAULT_CACERT="${VAULT_CACERT:-/opt/vault/tls/vault-ca.pem}"
BACKUP_DIR="${BACKUP_DIR:-/opt/vault/backups}"
MINIO_ALIAS="${MINIO_ALIAS:-myminio}"
MINIO_BUCKET="${MINIO_BUCKET:-vault}"
RETENTION_DAYS="${RETENTION_DAYS:-14}"
LOGFILE="${LOGFILE:-/var/log/vault-backup.log}"
MINIO_ACCESS_KEY=${MINIO_ACCESS_KEY:-$(cat /root/.accesskey)}
MINIO_SECRET_KEY=${MINIO_SECRET_KEY:-$(cat /root/.secretkey)}
###############################################################

export MC_CONFIG_DIR=/opt/vault/mc-config
mkdir -p $MC_CONFIG_DIR

mc alias set myminio http://192.168.64.4:9000 ${MINIO_ACCESS_KEY} ${MINIO_SECRET_KEY}

mkdir -p "$BACKUP_DIR"
touch "$LOGFILE"
# restrict log file
chmod 600 "$LOGFILE"

echo "$(date -u '+%F %T') INFO: starting vault backup" > "$LOGFILE"

# 1) Check that this node is active (leader).
HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" --cacert "$VAULT_CACERT" "${VAULT_ADDR}/v1/sys/health?standbyok=false" || echo "000")

if [ "$HTTP_CODE" -eq 200 ]; then
  echo "$(date) INFO: this node is active (HTTP 200)" >> "$LOGFILE"
elif [ "$HTTP_CODE" -eq 429 ]; then
  echo "$(date) INFO: this node is standby (HTTP 429) - exiting" >> "$LOGFILE"
  exit 1
elif [ "$HTTP_CODE" -eq 503 ]; then
  echo "$(date) ERROR: vault sealed or not initialized (HTTP 503) - exiting" >> "$LOGFILE"
  exit 1
else
  echo "$(date) ERROR: unexpected vault health HTTP code: $HTTP_CODE - exiting" >> "$LOGFILE"
  exit 1
fi

# 2) Create snapshot
TIMESTAMP=$(date +'%F-%H%M%S')
SNAPFILE="${BACKUP_DIR}/vault-${TIMESTAMP}.snap"
echo "$(date) INFO: creating snapshot $SNAPFILE" >> "$LOGFILE"

if vault operator raft snapshot save "$SNAPFILE"; then
  echo "$(date) INFO: snapshot created: $SNAPFILE" >> "$LOGFILE"
else
  echo "$(date) ERROR: snapshot creation failed" >> "$LOGFILE"
  exit 2
fi

# 3) Compress (saves bandwidth)
if gzip -f "$SNAPFILE"; then
  SNAPFILE="${SNAPFILE}.gz"
  echo "$(date) INFO: compressed snapshot to $SNAPFILE" >> "$LOGFILE"
else
  echo "$(date) WARN: gzip failed, continuing with uncompressed snap" >> "$LOGFILE"
fi

# 4) Upload to MinIO
echo "$(date) INFO: uploading $SNAPFILE to ${MINIO_ALIAS}/${MINIO_BUCKET}/" >> "$LOGFILE"
if mc cp "$SNAPFILE" "${MINIO_ALIAS}/${MINIO_BUCKET}/"; then
  echo "$(date) INFO: upload successful" >> "$LOGFILE"
else
  echo "$(date) ERROR: upload to MinIO failed" >> "$LOGFILE"
  exit 3
fi

# 5) Local retention - delete local files older than RETENTION_DAYS
echo "$(date) INFO: removing local snapshots older than ${RETENTION_DAYS} days" >> "$LOGFILE"
find "$BACKUP_DIR" -type f -name 'vault-*.snap*' -mtime +"$RETENTION_DAYS" -print -delete >> "$LOGFILE" 2>&1 || true

echo "$(date) INFO: vault backup completed" >> "$LOGFILE"
exit 0

Make it executable:

chmod +x /usr/local/bin/vault-backup-to-minio.sh

Systemd Unit: /etc/systemd/system/vault-backup.service

[Unit]
Description=Vault Raft Snapshot Backup to MinIO
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
ExecStart=/usr/local/bin/vault-backup-to-minio.sh
User=root
Group=root
# Protect system a bit more
ProtectSystem=full
ProtectHome=true
PrivateTmp=true
NoNewPrivileges=true

Timer: /etc/systemd/system/vault-backup.timer

[Unit]
Description=Run Vault backup script every 6 hours

[Timer]
OnCalendar=*:0/6  # every 6 hours on the hour
Persistent=true  # if missed while down, run at boot

[Install]
WantedBy=timers.target

Enable and start:

systemctl daemon-reload
systemctl enable --now vault-backup.timer

Step 6: Delete Secrets and Restore from MinIO

Delete the secret:

vault kv delete -versions=1 secret/demo
vault kv metadata delete secret/demo

Download snapshot:

mc cp myminio/vault-backups/vault-snapshots/vault-2025-10-06-064537.snap.gz /opt/vault/backups/

Unzip and restore:

gunzip vault-2025-10-06-064537.snap.gz
vault operator raft snapshot restore vault-2025-10-06-064537.snap

Verify the secret is restored:

vault kv get secret/demo

===== Secret Path =====
secret/data/demo

======= Metadata =======
Key                Value
---                -----
created_time       2025-10-04T07:42:19.123456Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

========== Data ==========
Key         Value
---         -----
event       HashiTalksAfrica
year        2025
location    Online

Conclusion

By combining Vault’s integrated storage (Raft) with MinIO, you will be able to build a highly available and resilient Vault setup—entirely with open-source tools and minimal cost.

This approach ensures that even if all Vault nodes fail, you can restore your data from a secure snapshot stored in MinIO, getting back to operational state in minutes. It’s a practical solution for teams running on-premises or in budget-conscious environments, where traditional enterprise Vault may not be feasible.

However, while this setup provides HA and data durability, there’s one more layer to make it truly production-ready:

Recommendation

Place a load balancer (such as Nginx, HAProxy, or an L4 balancer like Keepalived) in front of your Vault cluster.
The load balancer should:

Continuously check the health and leadership status of Vault nodes using the /v1/sys/health or /v1/sys/leader endpoint.

Forward client traffic only to the current leader node, since Vault accepts writes exclusively on the leader.

Automatically re-route requests to the new leader after failover, ensuring seamless continuity for applications and users.

This way, your Vault deployment achieves:

High Availability
Data Durability
Automatic Failover
Seamless Client Experience

With this foundation, you now have a production-grade, open-source Vault cluster that’s cost-effective, recoverable, and resilient against both node and data failures.

Using IaC (Terraform) to Deploy a FastAPI Microservice on EKS with for Datadog Monitoring

Salaudeen O. Abdulrasaq — Mon, 17 Mar 2025 21:42:27 +0000

Introduction

Modern cloud applications need scalability, observability, and automation. Manually deploying infrastructure is outdated; instead, we use Infrastructure as Code (IaC) for reproducibility and monitoring tools to keep track of application health.

In this blog, we’ll build and deploy a FastAPI microservice on Amazon EKS (Elastic Kubernetes Service) using Terraform, while integrating Datadog APM (Application Performance Monitoring) for real-time tracing and insights.

Why This Stack?

FastAPI – A lightweight, high-performance web framework for APIs.
EKS (Elastic Kubernetes Service) ☁️ – Managed Kubernetes for deploying containerized workloads.
Terraform 🏗️ – Automates infrastructure provisioning using declarative code.
Datadog 🐶 – Provides monitoring, logging, and distributed tracing for better observability.

Project Overview

The project is structured as follows:

eks-datadog-tracing/
│── .gitignore              # Git ignore file
│── .terraform.lock.hcl     # Terraform lock file
│── main.tf                 # Main Terraform configuration
│── outputs.tf              # Terraform outputs
│── providers.tf            # Terraform providers setup
│── README.md               # Project documentation
│── terraform.tfvars        # Terraform variable values
│── variables.tf            # Terraform variables
│
├── app/                    # Application source code
│   ├── Dockerfile          # Docker configuration for the application
│   ├── main.py             # FastAPI application source code
│   ├── requirements.txt    # Dependencies for the application
│
└── modules/                # Modular Terraform configurations
    ├── application/        # Application-specific module
    │   ├── main.tf         # Terraform configuration for the application module
    │   ├── outputs.tf      # Outputs for the application module
    │   ├── ReadMe.md       # Documentation for the module
    │   ├── variables.tf    # Variables for the module
    │
    ├── datadog/            # Datadog monitoring and logging module
    │   ├── datadog.tf      # Datadog configuration
    │   ├── datadog_dashboard.tf  # Datadog dashboards setup
    │   ├── datadog_metric.tf     # Datadog metrics configuration
    │   ├── outputs.tf      # Outputs for the Datadog module
    │   ├── variables.tf    # Variables for the Datadog module
    │
    ├── eks/                # EKS cluster module
    │   ├── main.tf         # Terraform configuration for EKS
    │   ├── outputs.tf      # Outputs for the EKS module
    │   ├── variables.tf    # Variables for the EKS module
    │
    ├── vpc/                # VPC networking module
    │   ├── main.tf         # Terraform configuration for VPC
    │   ├── outputs.tf      # Outputs for the VPC module
    │   ├── variables.tf    # Variables for the VPC module

How It Works

Terraform provisions an EKS cluster and networking infrastructure.
The FastAPI application is containerized with Docker and deployed to EKS.
Datadog APM is integrated to collect real-time traces and metrics.
Observability is improved with Datadog dashboards, metrics, and alerts.

Building the Infrastructure with Terraform
1️⃣ Provisioning the VPC
The VPC module creates the necessary networking resources

module "vpc" {
  source               = "./modules/vpc"
  vpc_name            = var.vpc_name
  vpc_cidr_block      = var.vpc_cidr_block
  vpc_private_subnets = var.vpc_private_subnets
  vpc_public_subnets  = var.vpc_public_subnets
}

2️⃣ Deploying the EKS Cluster
We use the EKS module to create a Kubernetes cluster:

module "eks" {
  source = "./modules/eks"

  k8s_name          = var.k8s_name
  vpc_id            = module.vpc.vpc_id
  subnet_ids        = module.vpc.private_subnet_ids
  cluster_version   = "1.24"
}

Run Terraform commands to deploy:

terraform init
terraform apply -auto-approve

Once completed, EKS is ready to deploy our FastAPI app.

Deploying the FastAPI App on EKS
1️⃣ Writing the FastAPI Application
Our FastAPI app (app/main.py) exposes several endpoints:

from fastapi import FastAPI
import time
import random
from ddtrace import tracer

app = FastAPI()

@tracer.wrap()
@app.get("/")
def hello():
    return {"message": "Hello, World!"}

@tracer.wrap()
@app.get("/slow")
def slow_function():
    time.sleep(2)
    return {"message": "This function is slow!"}

@tracer.wrap()
@app.get("/cpu-intensive")
def cpu_intensive():
    total = sum(i * i for i in range(10**6))
    return {"message": "CPU-intensive task completed!"}

2️⃣ Containerizing the Application
We package our app with a Dockerfile:

FROM python:3.9
WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . .
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "80"]

Build and push the image to a container registry:

docker build -t your-docker-repo/fastapi-app:latest .
docker push your-docker-repo/fastapi-app:latest

3️⃣ Deploying to Kubernetes
Create a Kubernetes deployment file (k8s/deployment.yaml):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: fastapi-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: fastapi
  template:
    metadata:
      labels:
        app: fastapi
    spec:
      containers:
        - name: fastapi
          image: your-docker-repo/fastapi-app:latest
          ports:
            - containerPort: 80

Apply the deployment:

kubectl apply -f k8s/deployment.yaml

Monitoring with Datadog
1️⃣ Setting Up Datadog APM
We configure Datadog in main.py:


from ddtrace import tracer
tracer.configure(
    hostname="datadog-agent.datadog",
    port=8126
)

2️⃣ Enabling Logs and Metrics
We define Datadog metrics in Terraform:

resource "datadog_monitor" "high_latency" {
  name    = "High Latency Alert"
  type    = "query alert"
  query   = "avg(last_5m):avg:trace.http.request.duration{service:fastapi-app} > 500"
  message = "Alert! API response time is too high!"
}

3️⃣ Viewing Metrics
Once deployed, log in to Datadog and navigate to:

APM > Services – View real-time traces.
Metrics > Dashboards – Monitor CPU, latency, and traffic.

Testing the API
To check if everything is running:

# Basic "Hello World"
curl http://your-app-url/

# Simulate slow responses
curl http://your-app-url/slow
curl http://your-app-url/random-delay

# Heavy load endpoints
curl http://your-app-url/cpu-intensive

You can also view Swagger UI:

http://your-app-url/docs

Conclusion
By following this setup, we successfully:
✅ Deployed FastAPI as a microservice
✅ Used Terraform to provision EKS and infrastructure
✅ Integrated Datadog for tracing, logging, and monitoring

This approach ensures scalability, observability, and automation, making it ideal for production environments.

If you're interested in extending this, you can:

Add autoscaling policies for Kubernetes pods
Implement Datadog alerts for anomaly detection
Enable log aggregation using Fluentd
feel free to contribute!

What's Next After Passing The AWS Cloud Practitioner and Solution Architect Exam

Salaudeen O. Abdulrasaq — Mon, 18 Nov 2024 07:14:52 +0000

Congratulations if you just passed your AWS Cloud Practitioner (CCP) and the AWS Solutions Architect Associate (SAA) exam.
Here’s a structured path to help you grow further and leverage your new skills effectively:

1. Share Your Knowledge Through Content Creation

Start creating content based on what you’ve learned in the program, no matter how simple or basic the topic may seem. Remember, there’s always a demand for beginner-friendly "Cloud 101" resources.

Ideas for Content: Write blog posts, create YouTube videos, or share threads on social media about AWS concepts, services, or technologies.

Benefits: This not only reinforces your learning but also helps build your online presence and credibility in the cloud space.

2. Join the AWS Community Builders Program

After building a content portfolio, join the AWS Community Builders Program waitlist.

Advantages of Being an AWS Community Builder:

Exclusive access to AWS training and resources.
Networking opportunities with cloud enthusiasts and industry professionals.
Recognition in the AWS community, which can boost your career.
Swags, AWS Credit, and Voucher to take other AWS Certification Exams.

How Your Content Helps: Having published content on AWS technologies demonstrates your passion and commitment, increasing your chances of being accepted into the program.

3. Work on Personal Projects

Apply your knowledge by building hands-on projects. Personal projects not only deepen your understanding but also make you stand out when applying for jobs.

Get Inspiration: Check out this YouTube video on project ideas to kickstart your journey.

Project Examples:

Create a simple web application hosted on AWS.

Implement a serverless architecture with AWS Lambda and API Gateway.

Set up an automated CI/CD pipeline using AWS CodePipeline.

4. Start Applying for Jobs

While working on projects, begin applying for roles such as Cloud Engineer, DevOps Engineer, or Solutions Architect. Highlight:

Your certifications.

Content you've created.

Hands-on project experience.

Using CloudFormation to deploy a web app with HA

Salaudeen O. Abdulrasaq — Sun, 17 Nov 2024 15:21:29 +0000

INTRODUCTION

In this post, I am thrilled to share an exciting project I had the opportunity to work on. I embarked on a journey that not only expanded my knowledge but also empowered me to apply cutting-edge cloud computing and DevOps practices. Specifically, this project delved into Infrastructure as Code with AWS CloudFormation, providing me with invaluable hands-on experience in building scalable cloud infrastructure.

STATEMENT OF PROBLEM

Scenario
Your company is creating an Instagram clone.

Developers want to deploy a new application to the AWS infrastructure.

You have been tasked with provisioning the required infrastructure and deploying a dummy application, along with the necessary supporting software.

This needs to be automated so that the infrastructure can be discarded as soon as the testing team finishes their tests and gathers their results.

Optional - To add more challenge to the project, once the project is completed, you can try deploying sample website files located in a public S3 Bucket to the Apache Web Server running on an EC2 instance.

Server specs

Launch Configuration was created for application servers in order to deploy four servers, two located in each of your private subnets. The launch configuration was used by an auto-scaling group. Two vCPUs were used with 4GB of RAM. The Operating System used is Ubuntu 18. An Instance size and Machine Image (AMI) that best fits this spec was chosen.

MY SOLUTION:

Architecture Diagram

Below is the content of the parameter files and configuration files for the network infrastructure, s3 buckets, and servers(EC2 instances)

Network Parameters

network.json

[
    {
    "ParameterKey": "EnvironmentName",
    "ParameterValue": "UdacityProject"
    },
    {
    "ParameterKey": "VPCCIDR",
    "ParameterValue": "10.0.0.0/16"
    },
    {
    "ParameterKey": "PubSubnet1CIDR",
    "ParameterValue": "10.0.1.0/24"
    },
    {
    "ParameterKey": "PubSubnet2CIDR",
    "ParameterValue": "10.0.2.0/24"
    },
    {
    "ParameterKey": "PrivSubnet1CIDR",
    "ParameterValue": "10.0.3.0/24"
    },
    {
    "ParameterKey": "PrivSubnet2CIDR",
    "ParameterValue": "10.0.4.0/24"
    }

    ]

S3 bucket Parameters

s3bucket.json

[{
    "ParameterKey": "EnvironmentName",
    "ParameterValue": "UdacityProject"
},
{
    "ParameterKey": "S3BucketName",
    "ParameterValue": "udacityprojects3webserverbucket"
}
]

Server (EC2 Instance) Parameters

servers.json

[
    {
    "ParameterKey": "EnvironmentName",
    "ParameterValue": "UdacityProject"
    }
]

Network Configuration

The network.yaml file below creates a network infrastructure with public and private subnets, routing, and internet access. It includes parameters for customizing the environment, VPC CIDR, and subnet CIDR blocks. The code creates resources such as VPC, internet gateway, subnets (public and private), NAT gateways, and route tables. Outputs are defined to export important values like VPC ID, route table IDs, and subnet IDs. This CloudFormation template enables the creation of a network setup suitable for routing internet traffic to both public and private subnets.

network.yaml

AWSTemplateFormatVersion: "2010-09-09"
Description: Creates the required network infrastructure for public and private routing with internet access
Parameters: 
  EnvironmentName: 
    Description: An Environment name that will be prefixed to resources
    Type: String
  VPCCIDR:
    Type: String
  PrivSubnet1CIDR:
    Type: String
  PrivSubnet2CIDR:
    Type: String
  PubSubnet1CIDR:
    Type: String
  PubSubnet2CIDR:
    Type: String

Resources:
  myVPC:
    Type: AWS::EC2::VPC
    Properties:
      CidrBlock: !Ref VPCCIDR
      EnableDnsHostnames: true
      EnableDnsHostnames: true
      Tags:
        - Key: Name
          Value: "MainVPC"

# Create Internet Gateway
  InternetGateway:
    Type: AWS::EC2::InternetGateway
    Properties:
      Tags:
        - Key: Name
          Value: !Ref EnvironmentName

# Attached the internet Gateway to myVPC
  InternetGatewayAttached:    
    Type: AWS::EC2::VPCGatewayAttachment
    Properties: 
      InternetGatewayId: !Ref InternetGateway
      VpcId: !Ref myVPC

# Creating Public and Private Subnets in the same availability zone(us-east-1a).
  PublicSubnet1:
    Type: AWS::EC2::Subnet
    Properties:
      AvailabilityZone: "us-east-1a"
      CidrBlock: !Ref PubSubnet1CIDR
      VpcId:
        Ref: myVPC
      MapPublicIpOnLaunch: true
      Tags:
      - Key: Name
        Value: !Sub ${EnvironmentName} Public Subnet (AZ1)

  PrivateSubnet1:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId:
        Ref: myVPC
      CidrBlock: !Ref PrivSubnet1CIDR
      MapPublicIpOnLaunch: false
      AvailabilityZone: "us-east-1a"
      Tags:
        - Key: Name
          Value: !Sub ${EnvironmentName} Private Subnet (AZ1)


# Creating Public and Private Subnets in the same availability zone(us-east-1b).
  PublicSubnet2:
    Type: AWS::EC2::Subnet
    Properties:
      AvailabilityZone: "us-east-1b"
      CidrBlock: !Ref PubSubnet2CIDR
      VpcId:
        Ref: myVPC
      MapPublicIpOnLaunch: true
      Tags:
      - Key: Name
        Value: !Sub ${EnvironmentName} Public Subnet (AZ2)

  PrivateSubnet2:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId:
        Ref: myVPC
      CidrBlock: !Ref PrivSubnet2CIDR
      MapPublicIpOnLaunch: false
      AvailabilityZone: "us-east-1b"
      Tags:
        - Key: Name
          Value: !Sub ${EnvironmentName} Private Subnet (AZ2)

# Elastic IP for the NATGateway in Subnet1
  EIP1:
    Type: AWS::EC2::EIP
    DependsOn: InternetGatewayAttached
    Properties:
        Domain: myVPC
        Tags:
        - Key: Name
          Value: "Elastic IP for our NATGateway1"
  EIP2:
    Type: AWS::EC2::EIP
    DependsOn: InternetGatewayAttached
    Properties:
        Domain: myVPC
        Tags:
        - Key: Name
          Value: "Elastic IP for our NATGateway2"

# Creating NAT gateway in publicsubnet1
  NAT1:
    Type: AWS::EC2::NatGateway
    Properties:
        AllocationId:
          Fn::GetAtt:
          - EIP1
          - AllocationId
        SubnetId: !Ref PublicSubnet1
        Tags:
        - Key: Name
          Value: "NAT to be used by servers in the private subnet"
  NAT2:
    Type: AWS::EC2::NatGateway
    Properties:
        AllocationId:
          Fn::GetAtt:
          - EIP2
          - AllocationId
        SubnetId: !Ref PublicSubnet2
        Tags:
        - Key: Name
          Value: "NAT to be used by servers in the private subnet"



#_______PUBLIC SUBNET________
# # Route Table for Public subnet
  PublicRouteTable:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId: !Ref myVPC
      Tags:
      - Key: Name
        Value: !Sub ${EnvironmentName} Public Routes

# Create Route for public Subnet 1 & 2 
  PublicRoute:
    Type: AWS::EC2::Route
    DependsOn: InternetGatewayAttached
    Properties:
        RouteTableId: !Ref PublicRouteTable
        DestinationCidrBlock: 0.0.0.0/0
        GatewayId:
          Ref: InternetGateway

# Associate Route Table to Public subnet 1 & 2
  AssociatePublicRoute:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties: 
      RouteTableId: 
        Ref: PublicRouteTable
      SubnetId: !Ref PublicSubnet1

  AssociatePublicRoute:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties: 
      RouteTableId: 
        Ref: PublicRouteTable
      SubnetId: !Ref PublicSubnet2


# ______PRIVATE SUBNET 1______
# Route Table for Private subnet1

  PrivateRouteTable1:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId: !Ref myVPC
      Tags:
      - Key: Name
        Value: !Sub ${EnvironmentName} Private Routes (AZ1)

# Create Route for Private subnet1
  PrivateRoute1:
    Type: AWS::EC2::Route
    Properties:
        RouteTableId: !Ref PrivateRouteTable1
        DestinationCidrBlock: 0.0.0.0/0
        NatGatewayId:
          Ref: NAT1

# Private Route Table1 Association to Private Subnet1
  AssociatePrivateRoute:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties: 
      RouteTableId: !Ref PrivateRouteTable1
      SubnetId: !Ref PrivateSubnet1



# _____PRIVATE SUBNET 2_____
# Route Table for Private subnet2
  PrivateRouteTable2:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId: !Ref myVPC
      Tags:
      - Key: Name
        Value: !Sub ${EnvironmentName} Private Routes (AZ2)

# Create Route for Private subnet2
  PrivateRoute2:
    Type: AWS::EC2::Route
    Properties:
        RouteTableId: !Ref PrivateRouteTable2
        DestinationCidrBlock: 0.0.0.0/0
        NatGatewayId:
          Ref: NAT2

# Private Route Table2 Association to Private Subnet2
  AssociatePrivateRoute:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties: 
      RouteTableId: !Ref PrivateRouteTable2
      SubnetId: !Ref PrivateSubnet2


Outputs:
  myVPC:
    Description: The VPC created for this project
    Value: !Ref myVPC
    Export:
      Name: !Sub ${EnvironmentName}-VPCID

  PublicRouteTable: 
        Description: Public Route Table
        Value: !Ref PublicRouteTable
        Export:
          Name: !Sub ${EnvironmentName}-PUB-RT

  PrivateRouteTable1: 
        Description: Private route Table1
        Value: !Ref PrivateRouteTable1
        Export:
          Name: !Sub ${EnvironmentName}-PRI-RT1

  PrivateRouteTable2: 
        Description: Private route Table2
        Value: !Ref PrivateRouteTable2
        Export:
          Name: !Sub ${EnvironmentName}-PRI-RT2

  PublicSubnets:
        Description: A list of the public subnets
        Value: !Join [ ",", [ !Ref PublicSubnet1, !Ref PublicSubnet2 ]]
        Export:
          Name: !Sub ${EnvironmentName}-PUB-SUBNETS

  PublicSubnet1:
        Description: public subnet 1 in "us-east-1a"
        Value: !Ref PublicSubnet1
        Export:
          Name: !Sub ${EnvironmentName}-PUB-SUB1

  PublicSubnet2:
        Description: public subnet 2 in us-east-1b
        Value: !Ref PublicSubnet2
        Export:
          Name: !Sub ${EnvironmentName}-PUB-SUB2

  PrivateSubnets:
        Description: A list of the private subnets
        Value: !Join [ ",", [ !Ref PrivateSubnet1, !Ref PrivateSubnet2 ]]
        Export:
          Name: !Sub ${EnvironmentName}-PRIV-SUBNETS

  PrivateSubnet1:
        Description: private subnet 1 in us-east-1a
        Value: !Ref PrivateSubnet1
        Export:
          Name: !Sub ${EnvironmentName}-PRIV-SUB1

  PrivateSubnet2:
        Description: private subnet 1 in us-east-1b
        Value: !Ref PrivateSubnet2
        Export:
          Name: !Sub ${EnvironmentName}-PRIV-SUB2

  VPCdefaultSecurityGroup:
        Description: Returns the default security group of the created VPC
        Value: !GetAtt myVPC.DefaultSecurityGroup
        Export:
          Name: !Sub ${EnvironmentName}-myVPC-SG

S3 Bucket Configuration

The s3bucket.yaml file below creates an S3 bucket for deploying a high-availability web app. The bucket is configured with public read access, an index document, and an error document. A bucket policy allows all actions on the bucket, and an IAM role with AmazonS3FullAccess policy is created to enable EC2 instances to manage the web app. Outputs include the IAM role, website URL, and secure website URL. This CloudFormation template facilitates the setup of an S3 bucket for hosting a high-availability web app with appropriate permissions and URL accessibility.

s3bucket.yaml

Description: >
  Create an S3 bucket for deploying a high-availability web-app.


Parameters:
  EnvironmentName:
    Description: An environment name that will be prefixed to resource names.
    Type: String

  S3BucketName:
    Description: S3 bucket name.
    Type: String


Resources:
  S3WebServer:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: !Ref S3BucketName
      AccessControl: PublicRead
      WebsiteConfiguration:
        IndexDocument: index.html
        ErrorDocument: error.html
      Tags: 
        - Key: Name
          Value: !Sub ${EnvironmentName} s3webserver bucket
    DeletionPolicy: Delete

  S3WebAppPolicy:
    Type: AWS::S3::BucketPolicy
    Properties:
      Bucket: !Ref S3WebServer
      PolicyDocument:
        Statement:
          - Effect: Allow
            Action: s3:*
            Resource: !Join ['', ['arn:aws:s3:::', !Ref 'S3WebServer', '/*']]
            Principal:
              AWS: '*'

  WebServerIAMRole:
    Type: 'AWS::IAM::Role'
    Properties:
      ManagedPolicyArns:
        - 'arn:aws:iam::aws:policy/AmazonS3FullAccess'
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: 'Allow'
            Principal:
              Service:
                - 'ec2.amazonaws.com'
            Action:
              - 'sts:AssumeRole'
      Path: '/'

  MyInstanceProfile: 
    Type: "AWS::IAM::InstanceProfile"
    Properties: 
      Path: "/"
      Roles: 
        - 
          Ref: "WebServerIAMRole"



Outputs:

  WebServerIAMRole:
    Description: 'Allow EC2 instances to manage Web App S3'
    Value: !Ref MyInstanceProfile
    Export:
      Name: !Sub ${EnvironmentName}-IAM-NAME

  # WebServerIAMRole:
  #   Description: Iam Instance Profile Arn
  #   Value: !GetAtt WebServerIAMRole.Arn
  #   Export:
  #     Name: !Sub ${EnvironmentName}-IAM-ARN

  WebsiteURL:
    Value: !GetAtt [S3WebServer, WebsiteURL]
    Description: URL for website hosted on S3
  WebsiteSecureURL:
    Value: !Join ['', ['https://', !GetAtt [S3WebServer, DomainName]]]
    Description: Secure URL for website hosted on S3

Server (EC2 instance) Configuration

The servers.yaml file below creates a network infrastructure with servers for hosting a high-availability web app.
The code defines several resources, including security groups, launch configuration, auto scaling group, load balancer, listener, target group, scaling policies, and outputs. These resources enable the setup of a load-balanced environment for the web app.
The code provides an output called LoadBalancerEndpoint, which represents the endpoint for reaching the load balancer.
Overall, this CloudFormation template facilitates the creation of a load-balanced infrastructure with auto scaling capabilities for hosting web servers.

servers.yaml

AWSTemplateFormatVersion: "2010-09-09"
Description: Creates the required servers in the network infrastructure defined by "network.yml"
Parameters: 
  EnvironmentName: 
    Description: An Environment name that will be prefixed to resources
    Type: String

  InstanceType:
    Description: Amazon EC2 instance type for the instances
    Type: String
    AllowedValues:
      - t2.micro
      - t3.micro
      - t3.small
      - t3.medium
    Default: t2.micro

Mappings:
      WebServerRegion:
        us-east-1:
          HVM64: ami-052efd3df9dad4825


Resources:

#Loadbalancer security group.
  LBSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Allow http request to loadbalancer
      VpcId: 
        Fn::ImportValue:
          !Sub "${EnvironmentName}-VPCID"
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: 80
          ToPort: 80
          CidrIp: 0.0.0.0/0
      SecurityGroupEgress:
        - IpProtocol: tcp
          FromPort: 80
          ToPort: 80
          CidrIp: 0.0.0.0/0

# Web Server security group.
  InstanceSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Allow http to Webserver
      VpcId: 
        Fn::ImportValue:
          !Sub "${EnvironmentName}-VPCID"
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: 80
          ToPort: 80
          CidrIp: 0.0.0.0/0
        - IpProtocol: tcp
          FromPort: 22
          ToPort: 22
          CidrIp: 0.0.0.0/0
      SecurityGroupEgress:
        - IpProtocol: tcp
          FromPort: 0
          ToPort: 65535
          CidrIp: 0.0.0.0/0

  WebServerLaunchConfig:
    Type: AWS::AutoScaling::LaunchConfiguration
    Properties:
      IamInstanceProfile:
        Fn::ImportValue: !Sub '${EnvironmentName}-IAM-NAME'
      UserData: 
          # wget -P /var/www/html https://project2udacity.s3-us-west-2.amazonaws.com/index.html
        Fn::Base64: !Sub |
          #!/bin/bash
          apt-get update -y
          apt-get install unzip awscli -y
          apt-get install apache2 -y
          systemctl start apache2.service
          sudo rm /var/www/html/index.html
          sudo aws s3 cp s3://udacityprojects3webserverbucket/udagram.zip /var/www/html
          sudo unzip /var/www/html/udagram.zip -d /var/www/html
          sudo rm /var/www/html/udagram.zip 
          systemctl restart apache2.service
      ImageId: !FindInMap [WebServerRegion, !Ref 'AWS::Region', HVM64]
      SecurityGroups: 
        - !Ref InstanceSecurityGroup
      InstanceType: 
        !Ref InstanceType
      BlockDeviceMappings: 
        - DeviceName: /dev/sda1
          Ebs: 
            VolumeSize: '10'
            VolumeType: 'gp2'

  WebServerASG:
    Type: AWS::AutoScaling::AutoScalingGroup
    Properties:
      VPCZoneIdentifier:
      - Fn::ImportValue: !Sub "${EnvironmentName}-PRIV-SUB1"
      - Fn::ImportValue: !Sub "${EnvironmentName}-PRIV-SUB2"
      LaunchConfigurationName: !Ref WebServerLaunchConfig
      MaxSize: '4'
      MinSize: '4'
      DesiredCapacity: '4'
      TargetGroupARNs:
      - Ref: WebServerTargetGroup

  WebServerloadBalancer:
    Type: AWS::ElasticLoadBalancingV2::LoadBalancer
    Properties:
      Subnets:
        - Fn::ImportValue: !Sub "${EnvironmentName}-PUB-SUB1"
        - Fn::ImportValue: !Sub "${EnvironmentName}-PUB-SUB2"
      SecurityGroups:
      - Ref: LBSecurityGroup

  Listener:
    Type: AWS::ElasticLoadBalancingV2::Listener
    Properties:
      DefaultActions:
      - Type: forward
        TargetGroupArn:
          Ref: WebServerTargetGroup
      LoadBalancerArn:
        Ref: WebServerloadBalancer
      Port: '80'
      Protocol: HTTP

  ALBListenerRule:
    Type: AWS::ElasticLoadBalancingV2::ListenerRule
    Properties:
      Actions:
      - Type: forward
        TargetGroupArn:
          Ref: WebServerTargetGroup
      Conditions:
      - Field: path-pattern
        Values: [/]
      ListenerArn:
        Ref: Listener
      Priority: 1

  WebServerTargetGroup:
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
    Properties:
      HealthCheckIntervalSeconds: 5
      HealthCheckPath: /
      HealthCheckProtocol: HTTP
      HealthCheckTimeoutSeconds: 4
      HealthyThresholdCount: 3
      Port: 80
      Protocol: HTTP
      UnhealthyThresholdCount: 3
      VpcId:
        Fn::ImportValue:
          Fn::Sub: "${EnvironmentName}-VPCID"

# Scaling Policy Description (Up)
  WebServerScaleDown:
    Type: AWS::AutoScaling::ScalingPolicy
    Properties:
      AdjustmentType: ChangeInCapacity
      AutoScalingGroupName: !Ref WebServerASG
      Cooldown: 300
      ScalingAdjustment: 1

# Scaling Policy Description (Down)
  WebServerScaleDown:
    Type: AWS::AutoScaling::ScalingPolicy
    Properties:
      AdjustmentType: ChangeInCapacity
      AutoScalingGroupName: !Ref WebServerASG
      Cooldown: 300
      ScalingAdjustment: -1

Outputs:
    LoadBanlancerEndpoint: 
        Description: this endpoint is used to reach the loadbalancer.
        Value: !Join [ "", [ 'http://', !GetAtt WebServerloadBalancer.DNSName  ]]
        Export:
          Name: !Sub ${EnvironmentName}-LBENDPOINT

Script Usage
This Repository contains some scripts that will be used to create the CloudFormation stack.

Usage:
Create:
./create.sh (stackName) (script.yml) (parameters.json) (profile)
Example:

./create.sh Udagram infrastructure/network.yaml parameters/network.json udacity_user

Below is the content of the create.sh script

aws cloudformation create-stack --stack-name $1 --template-body file://$2 --parameters file://$3 --capabilities "CAPABILITY_IAM" "CAPABILITY_NAMED_IAM" --region=us-east-1 --profile=$4

Update:
./update.sh (stackName) (script.yml) (parameters.json) (profile)
Example:

./update.sh Udagram infrastructure/network.yaml parameters/network.json udacity_user

Below is the content of the update.sh script

aws cloudformation update-stack --stack-name $1 --template-body file://$2 --parameters file://$3 --capabilities "CAPABILITY_IAM" "CAPABILITY_NAMED_IAM" --region=us-east-1 --profile=$4

Nutanix Hyperconverged Infrastructure: Bridging the Gap Between On-Premise and Public Cloud

Salaudeen O. Abdulrasaq — Wed, 23 Oct 2024 18:11:02 +0000

Introduction

A concise overview of Nutanix HCI and the idea of Hyperconverged Infrastructure.
Discussion on the significance of HCI in contemporary enterprise IT, particularly in relation to digital transformation.
Transition into how Nutanix HCI corresponds with the services and scalability provided by public clouds.

What is Nutanix Hyperconverged Infrastructure (HCI)?

Definition and description of HCI: a unified software-defined platform that combines compute, storage, and networking.
Explanation of how Nutanix HCI streamlines data center management by integrating resources.

Advantages

Easier management
Scalability
Cost-effectiveness, and
Adaptability.

Nutanix HCI vs. Public Cloud

Elasticity & Scalability: Nutanix HCI can scale similarly to public clouds (e.g., AWS, Azure, Google Cloud) to accommodate increasing workloads.
Distinctions between horizontal and vertical scaling.
Deployment Flexibility: Nutanix provides hybrid and multi-cloud solutions, enabling organizations to deploy workloads both on-premises and across public clouds seamlessly.
Comparison with public cloud services that operate on shared infrastructure, while Nutanix gives a private cloud-like level of control.
Resource Management: Automated resource allocation and infrastructure oversight on both platforms.
Public cloud services usually charge based on usage, while Nutanix offers cost management through predictable pricing structures.

Nutanix Services Available on HCI

Nutanix Acropolis (AOS): The foundational element of Nutanix HCI that delivers VM-centric management, integrated backup, disaster recovery, and data protection.

Nutanix Prism: A centralized management platform for monitoring and managing clusters, akin to cloud dashboards for resource oversight.

Nutanix Calm: Facilitates automation and orchestration of applications across various environments, offering features comparable to cloud automation tools (e.g., AWS CloudFormation or Azure Resource Manager).

Nutanix Files: A scalable file storage solution that provides capabilities similar to services like AWS EFS or Azure Files.

Nutanix Objects: Object storage within HCI, parallel to offerings like AWS S3 or Google Cloud Storage.

Nutanix Era: Database management and provisioning, similar to database services like AWS RDS or Azure SQL Database.

Nutanix Karbon: Kubernetes orchestration for containerized applications, delivering cloud-native capabilities on-premises, akin to EKS or GKE.

Seamless Hybrid and Multi-Cloud Integration

Nutanix's capability to integrate with public clouds via Nutanix Clusters, allowing users to shift workloads between Nutanix HCI and AWS or Azure.
Benefits of maintaining both on-premises and cloud environments using Nutanix as a unified infrastructure.

Conclusion

Recap of how Nutanix HCI connects traditional data centers with public clouds.
Final reflections on the future of Nutanix HCI in hybrid cloud deployments and its potential to simplify cloud operations while providing enterprises with the control and flexibility they require.

"For further details on integration options, check the Nutanix Documentation."

Automate Uploading Security Scan Results to DefectDojo

Salaudeen O. Abdulrasaq — Sun, 15 Sep 2024 18:45:14 +0000

In my previous blog, I explored secret scanning in CI pipelines using Gitleaks and pre-commit hooks. Beyond the risk of exposing secrets in repositories and pipelines, vulnerabilities can also be introduced into an application through third-party libraries and dependencies.

The first step in addressing and managing these vulnerabilities is to accurately collect, categorize, and present them in a readable format.

A powerful tool for managing security vulnerabilities is DefectDojo, which helps streamline the vulnerability management process.

"DefectDojo is a security orchestration and vulnerability management platform. DefectDojo allows you to manage your application security program, maintain product and application information, triage vulnerabilities, and push findings to systems like JIRA and Slack. DefectDojo enriches and refines vulnerability data using a number of heuristic algorithms that improve the more you use the platform."
— Source: DefectDojo DockerHub page

Overview

The process involves utilizing a Python script (upload.py) that interfaces with the DefectDojo API to import scan results from various security tools such as Gitleaks, Semgrep, and NJSSCAN. By integrating this script into GitLab CI/CD pipeline from here, you can automate the process of uploading scan results whenever a new scan is performed.

Demo Server

To simplify the process of setting up a DefectDojo instance, you can try out the demo server at demo.defectdojo.org. Log in with:

Username: admin
Password: 1Defectdojo@demo#appsec

Please note that the demo is publicly accessible and regularly reset. Do not put sensitive data in the demo.

When managing security vulnerabilities in an organization, it's crucial to have a structured approach to tracking, categorizing, and remediating issues. DefectDojo provides this structure through a set of well-defined concepts and terms that facilitate the organization and management of security-related activities.

Key Terms and Concepts in DefectDojo

Report: After testing, DefectDojo enables you to generate detailed reports on the findings. These reports are essential for communicating security issues to stakeholders.

Product: This represents the application or system you're securing. Each product can have different versions or components, allowing you to keep track of the security status of multiple versions.

Product Type: Products are grouped into categories called product types. For example, you might categorize products as web applications, APIs, or microservices. This helps in reporting and filtering across multiple similar products.

Engagement: DefectDojo organizes security assessments through engagements. Each engagement represents a security testing activity—such as a penetration test or vulnerability scan—on a specific product.

Test: Tests are actions carried out within an engagement, like running a security tool or performing manual code reviews. Each engagement can contain multiple tests.

Finding: Findings are the vulnerabilities or issues discovered during a test. These are tracked, prioritized, and assigned for remediation, and DefectDojo's heuristic algorithms help refine and enrich this data for better decision-making.

Endpoint: Products often contain multiple IP addresses, URLs, or domains that need testing. Endpoints allow you to track which specific parts of your product were tested.

Risk Acceptance: Not all vulnerabilities can or need to be immediately fixed. In cases where the risk is acknowledged but deemed acceptable, you can mark it as a risk acceptance, ensuring proper documentation.

By understanding and utilizing these concepts, you can better manage your application security program and streamline your vulnerability management process using DefectDojo.

The Python Script

Firstly, generate API key as shown below:

This API Key will be used in the Python script to automate scan report upload.

Here’s a breakdown of the upload.py script:

import requests
import sys

file_name = sys.argv[1]
scan_type = ''

if file_name == 'gitleaks.json':
    scan_type = 'Gitleaks Scan'
elif file_name == 'njsscan.sarif':
    scan_type = 'SARIF'
elif file_name == 'semgrep.json':
    scan_type = 'Semgrep JSON Report'

headers = {
    'Authorization': 'Token 548afd6fab3bea9794a41b31da0e9404f733e222'
}

url = 'https://demo.defectdojo.org/api/v2/import-scan/'

data = {
    'active': True,
    'verified': True,
    'scan_type': scan_type,
    'minimum_severity': 'Low',
    'engagement': 19
}

files = {
    'file': open(file_name, 'rb')
}

response = requests.post(url, headers=headers, data=data, files=files)

if response.status_code == 201:
    print('Scan results imported successfully')
else:
    print(f'Failed to import scan results: {response.content}')

Key Components

File Handling: The script accepts a filename as an argument, determining the type of scan based on the file extension.
API Integration: It sends a POST request to the DefectDojo API with the necessary headers and data, including the scan type and engagement ID.
Response Handling: It checks the response status to confirm whether the upload was successful.

Security Considerations

Ensure that the API token used in the script has the necessary permissions and is kept secure. Avoid hardcoding sensitive information; consider using environment variables or secret management tools.

GitLab CI/CD Integration

To automate the execution of this script, you can create a .gitlab-ci.yml file that defines the stages of your CI/CD pipeline. Here’s an example configuration:

variables:
  IMAGE_NAME: sirlawdin/demo-app
  IMAGE_TAG: juice-shop-1.1

stages:
  - cache
  - test
  - build

create_cache:
  image: node:18-bullseye
  stage: cache
  script:
    - yarn install --ignore-engines
  cache:
    key:
      files:
        - yarn.lock
    paths:
      - node_modules/
      - yarn.lock
      - .yarn
    policy: pull-push

yarn_test:
  image: node:18-bullseye
  stage: test
  script:
    - yarn install
    - yarn test
  cache:
    key:
      files:
        - yarn.lock
    paths:
      - node_modules/
      - yarn.lock
      - .yarn
    policy: pull-push

gitleaks:
  stage: test
  image:
    name: ghcr.io/gitleaks/gitleaks:latest
    entrypoint: [""]
  script:
    - gitleaks detect --source . --verbose --format json --report-path gitleaks-report.json
  allow_failure: true
  artifacts:
    when: always
    paths:
      - gitleaks-report.json

njsscan:
  stage: test
  image: python:3.11-slim
  before_script:
    - pip install --upgrade njsscan
  script:
    - njsscan --exit-warning . --sarif -o njsscan.sarif
  allow_failure: true
  artifacts:
    when: always
    paths:
      - njsscan.sarif

semgrep:
  stage: test
  image: returntocorp/semgrep:latest
  variables:
    SEMGREP_RULES: p/javascript
  script:
    - semgrep ci --json --output semgrep.json
  allow_failure: true
  artifacts:
    when: always
    paths:
      - semgrep.json

upload_reports:
  stage: test
  image: python
  needs: ["gitleaks", "njsscan", "semgrep"]
  when: always
  before_script:
    - pip3 install requests
  script:
    - python3 upload-reports.py gitleaks.json
    - python3 upload-reports.py njsscan.sarif
    - python3 upload-reports.py semgrep.json


build_image:
  stage: build
  image: docker:24
  services:
    - docker:24-dind
  before_script:
    - echo $DOCKER_PASS | docker login -u $DOCKER_USER --password-stdin
  script:
    - docker build -t $IMAGE_NAME:$IMAGE_TAG .
    - docker push $IMAGE_NAME:$IMAGE_TAG

include:
  - remote: 'https://gitlab.com/gitlab-org/gitlab/-/raw/master/lib/gitlab/ci/templates/Jobs/Secret-Detection.latest.gitlab-ci.yml'

Explanation of the CI/CD Configuration

Stages: Define a stage named security_scan.
Script: Call the upload.py script for each scan result file generated during the CI process.

Conclusion

By automating the upload of security scan results to DefectDojo, you can streamline your vulnerability management workflow. This integration not only saves time but also ensures that security issues are promptly addressed. Implementing this solution within your CI/CD pipeline will enhance the overall security posture. To learn more about DefectDojo, you can use the documentation.

Upcoming AWS Service Deprecations!!!

Salaudeen O. Abdulrasaq — Fri, 02 Aug 2024 08:01:50 +0000

I do not like hearing about AWS deprecations, especially for services that I have used frequently over the years.

However, as AWS evolves, some services inevitably need to be deprecated. Here’s a brief overview of the notable ones:

AWS CodeCommit

Code Commit is the GitHub and GitLab of AWS with seamless integration with some AWS services. AWS CodeCommit has stopped onboarding new customers. From now on, a new repository can only be created by customers with an existing repository in AWS CodeCommit. While still functional, AWS may shift focus to other version control solutions. Users should evaluate alternatives that align with their development workflows.

AWS Cloud9

Cloud9 is an integrated development environment (IDE), similar to Replit, Gitpod, CodeSandbox, or a browser-based version of VS Code. Users should consider transitioning to different IDEs or local setups as AWS has announced deprecation. According to them
“After careful consideration, we have made the decision to close new customer access to AWS Cloud9, effective July 25, 2024. AWS Cloud9 existing customers can continue to use the service as normal. AWS continues to invest in security, availability, and performance improvements for AWS Cloud9, but we do not plan to introduce new features.” (source)

AWS Mobile Hub

This service simplifies the process of building, testing, and monitoring mobile applications and can make use of one or more AWS services. With the rise of more advanced mobile development platforms, AWS Mobile Hub is being deprecated. Developers are encouraged to migrate to AWS Amplify for enhanced mobile app development capabilities.

Amazon CloudSearch

Amazon CloudSearch is a fully managed service in the cloud that makes it easy to set up, manage, and scale a search solution for your website or application. As AWS introduces more powerful search solutions, Amazon CloudSearch may be phased out. Users should explore Amazon OpenSearch Service for advanced search functionalities.

AWS CodeStar

AWS CodeStar is a cloud‑based development service that provides the tools you need to quickly develop, build, and deploy applications on AWS. With AWS CodeStar, you can set up your entire continuous delivery toolchain in minutes, allowing you to start releasing code faster.
This service is expected to be deprecated as AWS announced the Discontinuation of AWS CodeStar support

Amazon Quantum Ledger Database (QLDB)

QLDB may see reduced usage as AWS focuses on other database solutions. AWS recently announced that new customers can no longer sign up for Amazon Quantum Ledger Database (QLDB), a managed service providing an immutable transaction log maintained by a central trusted authority. All existing databases will be shut down in one year, and current users are encouraged to migrate to Aurora PostgreSQL.

To ensure a smooth transition, stay updated on AWS announcements and plan migrations accordingly. Embracing newer services can enhance your development processes and infrastructure.

Secret Scanning in CI pipelines using Gitleaks and Pre-commit Hook.

Salaudeen O. Abdulrasaq — Tue, 16 Jul 2024 05:40:34 +0000

In today's development environment, maintaining the security of your code is as crucial as ensuring its functionality. One of the key aspects of security is managing and safeguarding your secrets, such as API keys, passwords, and tokens. Accidentally committing these secrets to your repository can lead to severe security breaches. Implementing secret scanning in your Continuous Integration (CI) pipelines is essential to mitigate this risk. This blog will guide you through setting up secret scanning in GitLab CI pipelines using Gitleaks and a pre-commit hook.

Why Secret Scanning?

Before diving into the technical setup, let's briefly discuss why secret scanning is important:

Prevent Data Leaks: Secrets embedded in the code can be easily exposed if not managed properly, leading to unauthorized access.
Compliance: Many organizations have compliance requirements that mandate the protection of sensitive data.
Early Detection: Scanning for secrets early in the CI pipeline helps catch issues before they make it to production.
Faster Time to Market through Accelerated Releases: Automated secret scanning ensures that security checks do not become bottlenecks in the release process, allowing for faster and more frequent releases.

Tools Used

Gitleaks: An open-source tool that scans your codebase for secrets, providing a comprehensive way to detect and prevent leaks.
Git Hooks: This is a git functionality. It's a way to run custom scripts when certain actions occur.
OWASP Juice Shop: The application code to be used for this demonstration. It is probably the most modern and sophisticated insecure web application. Juice Shop encompasses vulnerabilities from the entire OWASP Top Ten and many other security flaws in real-world applications!
Gitlab: A web-based DevOps lifecycle tool that provides a Git repository manager with features like source code management (SCM), continuous integration (CI), continuous deployment (CD), and more.

STEP 1:
Fork the OWASP Juice Shop Gitlab repository

Added .env file that contains some dummy secrets to the repo.

STEP 2:
Create a .gitlab-ci.yml file in the repository. This file is the configuration file that GitLab CI/CD uses to define the pipeline and its various stages, jobs, and actions.

variables:
    IMAGE_NAME: sirlawdin/juice-shop-app
    IMAGE_TAG: juice-shop-1.1

stages:
    - cache
    - test
    - build 

create_cache:
    image: node:18-bullseye
    stage: cache
    script:
      - yarn install
    cache:
      key:
        files:
          - yarn.lock
      paths:
        - node_modules/
        - yarn.lock
        - .yarn
      policy: pull-push

gitleaks:
  stage: test
  image:
    name: zricethezav/gitleaks
    entrypoint: [""]
  script: 
    - gitleaks detect --source . --verbose --report-path gitleaks-report.json

yarn_test:
    image: node:18-bullseye
    stage: test
    script: 
      - yarn install
      - yarn test
    cache:
      key:
        files:
          - yarn.lock
      paths:
        - node_modules/
        - yarn.lock
        - .yarn
      policy: pull-push

build_image:
    stage: build
    image: docker:24
    services:
      - docker:24-dind
    variables:
        DOCKER_USER: sirlawdin
    before_script:
      - echo $DOCKER_PASS | docker login -u $DOCKER_USER --password-stdin
    script:
      - docker build -t $IMAGE_NAME:$IMAGE_TAG .
      - docker push $IMAGE_NAME:$IMAGE_TAG

Here's a brief explanation of each section:

IMAGE_NAME and IMAGE_TAG: These variables define the name and tag of the Docker image that will be built and pushed.

variables:
    IMAGE_NAME: sirlawdin/juice-shop-app
    IMAGE_TAG: juice-shop-1.1

cache: Stage for caching dependencies.
test: Stage for running tests and performing secret scanning.
build: Stage for building and pushing the Docker image.

stages:
    - cache
    - test
    - build

create_cache: This job uses the node:18-bullseye image to install dependencies with Yarn.
Caching: The node_modules/, yarn.lock, and .yarn directories are cached based on the yarn.lock file to speed up future pipeline runs.

create_cache:
    image: node:18-bullseye
    stage: cache
    script:
      - yarn install
    cache:
      key:
        files:
          - yarn.lock
      paths:
        - node_modules/
        - yarn.lock
        - .yarn
      policy: pull-push

gitleaks: This job uses the zricethezav/gitleaks image to scan the source code for secrets.
Report: The results are saved to gitleaks-report.json.

gitleaks:
  stage: test
  image:
    name: zricethezav/gitleaks
    entrypoint: [""]
  script: 
    - gitleaks detect --source . --verbose --report-path gitleaks-report.json

yarn_test: This job installs dependencies and runs tests using the node:18-bullseye image.
Caching: Similar to the create_cache job, it caches dependencies to speed up future runs.

yarn_test:
    image: node:18-bullseye
    stage: test
    script: 
      - yarn install
      - yarn test
    cache:
      key:
        files:
          - yarn.lock
      paths:
        - node_modules/
        - yarn.lock
        - .yarn
      policy: pull-push

build_image: This job uses the docker:24 image and the Docker-in-Docker (dind) service to build and push the Docker image.
Docker Login: The DOCKER_PASS and DOCKER_USER variables are used to log in to Docker Hub.
Build and Push: The Docker image is built with the specified IMAGE_NAME and IMAGE_TAG, and then pushed to the Docker registry.

build_image:
    stage: build
    image: docker:24
    services:
      - docker:24-dind
    variables:
        DOCKER_USER: sirlawdin
    before_script:
      - echo $DOCKER_PASS | docker login -u $DOCKER_USER --password-stdin
    script:
      - docker build -t $IMAGE_NAME:$IMAGE_TAG .
      - docker push $IMAGE_NAME:$IMAGE_TAG

After running the pipeline, the job at the gitleak job failed due to leaks found in the repository.

PS: The dummy secrets previously added to the repository were all flagged.

Fetching changes with git depth set to 20...
Initialized empty Git repository in /builds/Sirlawdin/juice-shop/.git/
Created fresh repository.
Checking out 5c0f71b8 as detached HEAD (ref is main)...
Skipping Git submodules setup
$ git remote set-url origin "${CI_REPOSITORY_URL}"
Executing "step_script" stage of the job script
00:08
Using docker image sha256:7da57b8e4dc2b857722c7fe447673bd938d452e61acbeb652bf90a210385457a for zricethezav/gitleaks with digest zricethezav/gitleaks@sha256:75bdb2b2f4db213cde0b8295f13a88d6b333091bbfbf3012a4e083d00d31caba ...
$ gitleaks detect --verbose --source .
    ○
    │╲
    │ ○
    ○ ░
    ░    gitleaks
Finding:     API_KEY=T8DWKJKAWUQ27MDSL902DBJAL
USERNAME= JaneDoe
Secret:      T8DWKJKAWUQ27MDSL902DBJAL
RuleID:      generic-api-key
Entropy:     3.973661
File:        .env
Line:        1
Commit:      5c0f71b87695579ad0d6999d485cd7b659ae6be6
Author:      Salaudeen O. Abdulrasaq
Email:       salaudeen.abdulrasaq2008@gmail.com
Date:        2024-07-16T04:30:31Z
Fingerprint: 5c0f71b87695579ad0d6999d485cd7b659ae6be6:.env:generic-api-key:1
Finding:     SECRET_KEY=IFW2dSNA02JD7BDJ2BCJLA
Secret:      IFW2dSNA02JD7BDJ2BCJLA
RuleID:      generic-api-key
Entropy:     3.754442
File:        .env
Line:        4
Commit:      5c0f71b87695579ad0d6999d485cd7b659ae6be6
Author:      Salaudeen O. Abdulrasaq
Email:       salaudeen.abdulrasaq2008@gmail.com
Date:        2024-07-16T04:30:31Z
Fingerprint: 5c0f71b87695579ad0d6999d485cd7b659ae6be6:.env:generic-api-key:4
Finding:     ...2dSNA02JD7BDJ2BCJLA
ACCESS_KEY=ROHXMPGWQLAHJI92NJSD
Secret:      ROHXMPGWQLAHJI92NJSD
RuleID:      generic-api-key
Entropy:     4.121928
File:        .env
Line:        5
Commit:      5c0f71b87695579ad0d6999d485cd7b659ae6be6
Author:      Salaudeen O. Abdulrasaq
Email:       salaudeen.abdulrasaq2008@gmail.com
Date:        2024-07-16T04:30:31Z
Fingerprint: 5c0f71b87695579ad0d6999d485cd7b659ae6be6:.env:generic-api-key:5
2:09PM INF 33 commits scanned.
2:09PM INF scan completed in 7.61s
2:09PM WRN leaks found: 80
Cleaning up project directory and file based variables
00:00
ERROR: Job failed: exit code 1

PRE_COMMIT HOOK
Implementing a pre-commit hook to scan for potential leaks in your code can prevent sensitive credentials from being committed to the repository. This proactive measure ensures sensitive information remains secure and never makes it to the repository commits, mitigating the risk of exposure to attackers.

Git Hooks; There are various types of hooks available in Git, including:

Pre-commit
Pre-push
Pre-rebase
And more...

Pre-commit hook is fired when changes made to a repo is about to be committed.

To create a pre-commit hook:

Open your terminal and navigate to the root directory of your Git repository.
cd /path/to/your/repository
Create the pre-commit hook file:
Inside the .git/hooks directory, create a file named pre-commit.
vi .git/hooks/pre-commit

Enter the below script into the pre-commit file

#!/bin/sh

# Configuration
GITLEAKS_IMAGE="zricethezav/gitleaks"
HOST_FOLDER_TO_SCAN="juice-shop-repo"
CONTAINER_FOLDER="/path"
REPORT_FILE="gitleaks-report.json"

# Function to display an error message and exit
exit_with_error() {
    echo "$1"
    exit 1
}

# Run Gitleaks
echo "Running Gitleaks for secret scanning..."
docker pull ${GITLEAKS_IMAGE} || exit_with_error "Failed to pull Gitleaks Docker image."

# Run Gitleaks container
docker run -v ${HOST_FOLDER_TO_SCAN}:${CONTAINER_FOLDER} ${GITLEAKS_IMAGE} detect --source="${CONTAINER_FOLDER}" --verbose --report-path=${REPORT_FILE}
GITLEAKS_EXIT_CODE=$?

# Check if Gitleaks detected any secrets
if [ ${GITLEAKS_EXIT_CODE} -ne 0 ]; then
    echo "Gitleaks detected secrets in your code. Please fix the issues before committing."
    cat ${REPORT_FILE}
    exit 1
fi

echo "Gitleaks found no secrets. Proceeding with commit."
exit 0

Explanation:

Configuration:
Defined variables for image name, host folder, container folder, and report file for easy configuration and readability.

Error Handling:
The function exit_with_error was added to handle errors gracefully and provide meaningful error messages.

Docker Pull:
Added error handling for the docker pull command to ensure the script exits if pulling the Gitleaks image fails.

Docker Run:
It is used to configure variables to run the Gitleaks container and capture its exit code.

Check Gitleaks Exit Code:
Checked the exit code of the Gitleaks run and displayed the report if secrets were detected, preventing the commit.

Make the hook executable; Change the file permissions to make the pre-commit hook executable.

chmod +x .git/hooks/pre-commit

The pre-commit hook has helped enforce secret protection by preventing commits until any leaked secrets are removed.

Now, your pre-commit hook will run every time you attempt to commit changes, helping to enforce code quality and security checks before the commit is finalized.

Introduction to Containerization on AWS ECS (Elastic Container Service) and Fargate

Salaudeen O. Abdulrasaq — Sun, 02 Jun 2024 14:24:15 +0000

Pre-requisite: To get the best from this series, it is expected that you understand how Docker works. You can use this link to get an overview of how docker works If you don't have an AWS account yet, you can follow the steps in my blog on How to create an AWS account to create one for the hands-on exercises.

This series will be released over the next couple of months. You are going to learn about the following concepts:

ECS (Elastic Container Service)
Fargate
Load Balancing
Auto Scaling
ECR (Elastic Container Registry)
CI/CD (Continuous Integration/Continuous Deployment)
Blue/Green Deployment
AWS X-Ray
Service Discovery
App Mesh

What is ECS and Fargate?

Amazon ECS is a fully managed container orchestration service that enables you to run and manage Docker containers at scale. With ECS, you can easily deploy applications in containers without needing to manage the underlying infrastructure. It integrates seamlessly with other AWS services, providing a comprehensive solution for running microservices, batch processes, and long-running applications.

AWS Fargate is a serverless compute engine for containers that work with both Amazon ECS (Elastic Container Service) and Amazon EKS (Elastic Kubernetes Service). Fargate eliminates the need to provision and manage servers, allowing you to specify and pay for resources per application. This makes it easier to build and deploy containerized applications without worrying about the underlying infrastructure.

ECS takes care of orchestrating your containerized applications, managing their lifecycle, and integrating with other AWS services.
Fargate provides compute power on-demand, automatically adjusting to the needs of your applications without requiring you to manage any servers.

This restaurant analogy below helps illustrate how ECS and Fargate work together to simplify the process of running containerized applications.

Understanding Amazon ECS and AWS Fargate with a Restaurant Analogy

Imagine you want to start a restaurant business. To do this, you need a physical restaurant location, kitchen equipment, chefs and staff to prepare and serve the food, and all the necessary supplies.

Amazon ECS (Elastic Container Service) is like a professional restaurant management company. They provide you with the restaurant space, organize the kitchen staff, handle the logistics of ordering supplies, and ensure everything runs smoothly. You tell them what kind of cuisine you want to serve, the menu, and any special requirements, and they take care of the rest.

ECS as the Restaurant Management Company: ECS handles the orchestration of your "menu items" (the containers), ensuring they are prepared, served, and maintained according to your needs. You don't need to worry about the details of managing the restaurant infrastructure.

Now, let's introduce AWS Fargate into the mix. Fargate is like having a magical, self-adjusting kitchen space. You simply describe your menu and the number of customers you expect, and the kitchen automatically scales to fit your needs, providing all the necessary cooking equipment and staff without any manual intervention.

Fargate as the Magical, Self-Adjusting Kitchen: With Fargate, you don't need to worry about the capacity of your kitchen or the logistics of managing the equipment and staff. You specify the requirements for your "menu items" (the containers), and Fargate automatically provides the necessary cooking resources. It's like having a kitchen that can expand or contract to perfectly accommodate your restaurant's needs, ensuring you always have the right amount of space and resources.

Putting It All Together
When you combine ECS and Fargate, it's like having the best of both worlds. You have a professional restaurant management company (ECS) handling the overall operations, and a magical, flexible kitchen space (Fargate) that adjusts to perfectly meet your restaurant's needs without any manual setup or management.

Now that you understand on a high level how ECS and Fargate work; let's move on to the next part of this blog series where you will learn how to Launch your first ECS container.

Up Next: Launch Your First ECS Container

How to Create a Free Tier AWS account

Salaudeen O. Abdulrasaq — Sun, 02 Jun 2024 10:59:34 +0000

Amazon Web Services (AWS) is one of the most popular cloud service providers in the world, offering a range of services from computing power to storage solutions. Creating an AWS account is the first step to accessing these powerful tools. In this blog, we'll guide you through the process of creating and logging into an AWS account, with step-by-step instructions and helpful GIFs to illustrate each part of the process.

Step 1: Create Your AWS Account

Creating an AWS account is a straightforward process. Follow these steps:

Visit the AWS Sign-Up Page:
Navigate to the AWS Sign-Up page and click on the "Create an AWS Account" button.

Enter Your Email and Choose a Password:
Enter a valid email address and choose a secure password. This email will be your root user email address, which has full access to all AWS services.

Choose an AWS Account Name:
Enter a unique AWS account name that will be associated with your account.

Verify Your Email:
AWS will send a verification email to the address you provided. Enter the verification code from the email to proceed.

Enter Your Contact Information:
Provide your contact details, including your full name, address, and phone number.

Choose Your Account Type:
Select either a personal or professional account based on your usage needs.

Enter Payment Information:
AWS requires a credit card or payment method on file. You won't be charged unless you use services beyond the AWS Free Tier limits.

Identity Verification:
AWS will verify your identity by sending a text message or automated phone call with a verification code.

Select a Support Plan:
Choose from various support plans based on your needs. The Basic plan is free and suitable for most new users.

Complete the Setup:
Review and complete the setup process. Your AWS account is now ready to use!

Step 2: Log into Your AWS Account

Once your account is set up, logging in is simple:

Go to the AWS Management Console.

Enter Your Root User Email:
Enter the email address you used to create your AWS account.

Enter Your Password:
Provide the password associated with your AWS account.

Using the root account for daily operations is not recommended due to security concerns. Instead, create an IAM user with administrative privileges.

Visit the AWS Management Console
Navigate to the IAM Service:
In the AWS Management Console, go to the IAM (Identity and Access Management) service.

Create a New User:

Click on "Users" in the left-hand menu.
Click the "Add user" button.
Set User Details:

Enter a username for the new user (e.g., admin-user).
Select "AWS Management Console access".
Set a custom password or choose to generate one automatically. Ensure the user is required to reset their password upon first login.
Assign Permissions:

Click "Next: Permissions".
Choose "Attach existing policies directly".
Select the "AdministratorAccess" policy.
Review and Create:

Review the user details and permissions.
Click "Create user".
Download Credentials:

Download the .csv file containing the user credentials or copy the access details. This file contains the login URL, username, and password for the new IAM user.

Network Policy in Kubernetes

Salaudeen O. Abdulrasaq — Sun, 02 Jun 2024 08:17:13 +0000

Secure communication between pods is critical in maintaining secure deployments. In this post, I will demonstrate how Kubernetes Network Policy can enforce fine-grained security controls in Kubernetes.

I will demonstrate how to set up and enforce network policies in a Minikube environment, ensuring a MYSQL pod in one namespace cannot be accessed by a client pod in another namespace after applying the policy.

Prerequisites

A working installation of Minikube
Basic Knowledge of Kubernetes concepts and resources
'kubectl' configured to interact with the Minikube cluster.

Start Minikube

setup the Kubernetes environment with Minikube
minikube start

Create Namespaces and Deploy Pods

Create two namespaces: database namespace; for the MySQL pod and client namespace; for the client pod connecting to the MYSQL Database.

Deploy a MYSQL pod in the 'database' namespace:

kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: mysql
  namespace: database
  labels:
    app: mysql
spec:
  containers:
  - name: mysql
    image: mysql:5.7
    env:
    - name: MYSQL_ROOT_PASSWORD
      value: password
EOF

Deploy a Client pod in the `client` namespace:

kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: client
  namespace: client
  labels:
    app: client
spec:
  containers:
  - name: client
    image: mysql:5.7
    command: ["sleep", "3600"]
EOF

Test Connectivity Before Apply Network Policy

Verify that the client pod can connect to the MYSQL pod:
kubectl exec -it client -n client -- sh

Connect to MySQL:
mysql -h <pod ip address> -u root -p

Implementing Kubernetes Network Policy

Now, we can create a Kubernetes Network Policy to deny access from the client namespace to the database namespace.

I prefer using the Cilium Kubernetes Network Policy Generator. This tool provides a user-friendly UI to interpret policies at a glance and create them in a few clicks. It can be used to develop Kubernetes Network policies and Cilium Network Policy

Cilium offers a more robust and feature-rich alternative to Kubernetes' built-in network policies, enabling advanced security features like deep packet inspection and layer 7 (Application Layer) policies.

Generate a Kubernetes Network Policy with Cilium Policy Generator

How to use the UI policy Generator

kubectl --apply -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-client-access
  namespace: database
spec:
  podSelector:
    matchLabels:
      app: mysql
  policyTypes:
    - Ingress
    - Egress
  ingress: []
  egress: []
EOF

Test Connectivity After Applying Network Policy

Verify that the client pod can no longer connect to the MySQL pod:
kubectl exec -it -n client -- sh
mysql -h <pod ip address> -u root -p

By implementing Kubernetes Network Policies, we can effectively control the communication between pods across namespaces, enhancing the security of our Kubernetes cluster. For more advanced and robust network policies, technologies like cilium can be used.

Forem: Salaudeen O. Abdulrasaq

AWS Security Groups Now Show Related Resources

Building Highly Available Vault on a Budget: Raft + MinIO for Resilience

Why This Approach?

Architecture Overview

Step 1: Prepare Your Environment

Step 2: Configure Vault for HA Using Raft Storage

Verify Cluster

Step 3: Enable KV Secrets Engine (Version 2)

Step 4: Automate Vault Snapshot Backups to MinIO

Step 5: Verify Cluster Backup

Step 6: Delete Secrets and Restore from MinIO

Conclusion

Recommendation

Using IaC (Terraform) to Deploy a FastAPI Microservice on EKS with for Datadog Monitoring

Introduction

Why This Stack?

Project Overview

How It Works

What's Next After Passing The AWS Cloud Practitioner and Solution Architect Exam

Using CloudFormation to deploy a web app with HA

INTRODUCTION

STATEMENT OF PROBLEM

Server specs

MY SOLUTION:

Architecture Diagram

Network Parameters

S3 bucket Parameters

Server (EC2 Instance) Parameters

Network Configuration

S3 Bucket Configuration

Server (EC2 instance) Configuration

Nutanix Hyperconverged Infrastructure: Bridging the Gap Between On-Premise and Public Cloud

Introduction

What is Nutanix Hyperconverged Infrastructure (HCI)?

Nutanix HCI vs. Public Cloud

Nutanix Services Available on HCI

Seamless Hybrid and Multi-Cloud Integration

Conclusion

Automate Uploading Security Scan Results to DefectDojo

Overview

Demo Server

Key Terms and Concepts in DefectDojo

The Python Script

Key Components

Security Considerations

GitLab CI/CD Integration

Explanation of the CI/CD Configuration

Conclusion

Upcoming AWS Service Deprecations!!!

AWS CodeCommit

AWS Cloud9

AWS Mobile Hub

Amazon CloudSearch

AWS CodeStar

Amazon Quantum Ledger Database (QLDB)

Secret Scanning in CI pipelines using Gitleaks and Pre-commit Hook.

Why Secret Scanning?

Tools Used

Introduction to Containerization on AWS ECS (Elastic Container Service) and Fargate

Understanding Amazon ECS and AWS Fargate with a Restaurant Analogy

How to Create a Free Tier AWS account

Step 1: Create Your AWS Account

Step 2: Log into Your AWS Account

Network Policy in Kubernetes

Prerequisites

Start Minikube

Create Namespaces and Deploy Pods

Deploy a MYSQL pod in the 'database' namespace:

Deploy a Client pod in the client namespace:

Test Connectivity Before Apply Network Policy

Implementing Kubernetes Network Policy

Generate a Kubernetes Network Policy with Cilium Policy Generator

Test Connectivity After Applying Network Policy

Deploy a Client pod in the `client` namespace: