Forem: Tirth T

[Boost]

Tirth T — Thu, 12 Mar 2026 11:04:28 +0000

I Moved My Database Behind a VPN on AWS — Here's Every Step (With the Networking Concepts That Actually Matter)

Tirth T ・ Mar 12

#aws #security #beginners #networking

I Moved My Database Behind a VPN on AWS — Here's Every Step (With the Networking Concepts That Actually Matter)

Tirth T — Thu, 12 Mar 2026 11:04:10 +0000

The Moment I Realized Our Database Was Naked

I was auditing our AWS security groups when I saw this:

Security Group: sg-0139b9d03b7d72f11
Inbound Rules:
  MySQL/Aurora  3306  0.0.0.0/0    ← THE ENTIRE INTERNET

Our production MySQL RDS — the one with real user data — was accessible from every IP address on the planet. The only thing standing between an attacker and our database was a username and password.

No VPN. No private subnet. No network-level isolation. Just vibes and a MySQL password.

If you're reading this and thinking "wait, let me check mine..." — yeah, go check. I'll wait.

What We're Building

By the end of this guide, your architecture goes from this:

BEFORE :

  Anyone on the Internet
       │
       │  Direct connection, port 3306 wide open
       ▼
  MySQL RDS (Public IP, 0.0.0.0/0)

To this:

AFTER :

  Developer Laptop
       │
       │ 1. Connect to VPN (encrypted tunnel)
       ▼
  OpenVPN Server (EC2, public subnet)
       │
       │ 2. NAT translates VPN IP → VPC IP
       ▼
  MySQL RDS (private subnet, NO public IP, NO internet route)
       ▲
       │ 3. Backend servers connect directly (same VPC)
  Elastic Beanstalk / ECS / EC2

What you'll learn:

What actually makes a subnet "private" (it's one missing route)
How PKI (Public Key Infrastructure) works — the certificate system behind HTTPS, Kubernetes, and now your VPN
How Linux routes packets between networks (IP forwarding + NAT)
How to set up OpenVPN from scratch on Amazon Linux 2023
Every command explained — no "just run this and trust me"

What you'll need:

An AWS account
An existing VPC (we'll use the default one — you already have it)
~30 minutes
Basic terminal comfort

Why OpenVPN? (And Not Tailscale or WireGuard)

Quick honest comparison before we start:

	OpenVPN	WireGuard	Tailscale
Setup time	1-2 hr	30 min	5 min
Protocol	TCP + UDP	UDP only	UDP only
Auth model	Full PKI (certificates)	Simple key pairs	SSO/OAuth
Learning value	Very high	High	Low
What you own	Everything	Everything	Tailscale owns the coordination

I chose OpenVPN because:

TCP support — WireGuard is UDP-only. Some corporate networks and hotel WiFi block non-standard UDP. OpenVPN over TCP on port 443 looks like HTTPS traffic to firewalls. It Just Works everywhere.
PKI is a career skill — The certificate infrastructure you'll build here is the same concept behind Kubernetes mTLS, HTTPS, zero-trust architectures, and enterprise security. Understanding it once unlocks understanding everywhere.
Full ownership — No third-party coordination server. No account. No vendor. Just your CA, your certs, your server.

If you want "it works in 5 minutes and I don't care how," use Tailscale. Seriously, it's great. But if you want to understand networking, keep reading.

Phase 1: Build the Private Network Layer

The One Thing That Makes a Subnet Private

Here's the secret nobody explains clearly: a subnet is private because its route table doesn't have a route to the Internet Gateway. That's it. That's the whole thing.

Public subnet route table:
  172.31.0.0/16  → local        (VPC internal traffic)
  0.0.0.0/0      → igw-xxxxx    (everything else → internet)  ← THIS makes it public

Private subnet route table:
  172.31.0.0/16  → local        (VPC internal traffic)
                                  ← NO igw route = private

No magic. No special "private" checkbox on the subnet itself. Just a missing route.

The local route means "any traffic for 172.31.x.x stays inside the VPC." Since there's no route for anything else (0.0.0.0/0), packets to/from the internet simply have nowhere to go. But — critically — your backend servers in other VPC subnets CAN still reach this private subnet because of that local route.

Step 1.1: Create Two Private Subnets

Why two? RDS requires a "subnet group" spanning at least 2 Availability Zones, even for single-AZ deployments. It's an AWS requirement.

Go to VPC Console → Subnets → Create subnet:

Subnet A:

VPC:               Your VPC (e.g., vpc-0d8e4d1.....)
Name:              private-db-1a
Availability Zone: ap-south-1a (or your preferred AZ)
IPv4 CIDR:         172.31.48.0/20

Subnet B:

VPC:               Same VPC
Name:              private-db-1b
Availability Zone: ap-south-1b
IPv4 CIDR:         172.31.64.0/20

Choosing the CIDR: If your existing subnets use .0, .16, and .32 ranges (default VPC), the next available /20 blocks are .48 and .64. Each /20 gives you 4,096 IPs — way more than needed, but it's the standard block size.

For both subnets, go to Actions → Edit subnet settings and make sure "Enable auto-assign public IPv4 address" is unchecked. Belt and suspenders.

Step 1.2: Create the Private Route Table

VPC Console → Route tables → Create route table:

Name: rt-private-db
VPC:  Your VPC

After creation, verify the Routes tab shows ONLY:

Destination      Target
172.31.0.0/16    local

No 0.0.0.0/0 → igw-xxxxx. That's the whole point.

Step 1.3: Associate Your Subnets

On the route table page → Subnet associations → Edit → Check ONLY your two private subnets → Save.

Without this, your subnets default to the Main route table (which has the IGW route, making them public). Explicit association is required.

Step 1.4: Create the RDS Subnet Group

RDS Console → Subnet groups → Create:

Name:        my-private-db-subnet
Description: Private subnets for RDS - no internet access
VPC:         Your VPC
AZs:         Select both AZs
Subnets:     Select ONLY your two private subnets

Step 1.5: Launch Your RDS in the Private Subnet

RDS Console → Create database:

The key settings:

Setting	Value	Why it matters
DB subnet group	`my-private-db-subnet`	This is THE setting — places RDS in private subnets
Public access	No	No public IP assigned
VPC security group	Create new: `private-db-sg`	We'll configure this next

For everything else (engine, instance class, storage) — use whatever matches your needs. I'm using MySQL 8.0 on db.t3.micro for this guide.

Step 1.6: Configure the DB Security Group

This is where you define WHO can talk to your database. Edit the inbound rules of private-db-sg:

Port	Source	Why
3306	Your backend's security group ID	Backend servers connect via SG reference
3306	VPN server's security group ID	Developer access (we'll create this SG in Phase 2)

Why reference security groups instead of IPs? Because your backend IPs can change (scaling events, deployments, instance replacements). The security group stays the same. It's the AWS-native way to say "allow traffic from any EC2 that has this role."

Notice what's NOT here: No 0.0.0.0/0. No random developer IPs. No "I'll fix it later" rules.

Verify: Can You Reach the Database?

From your laptop (no VPN yet):

# Windows:
Test-NetConnection -ComputerName your-db.xxxx.region.rds.amazonaws.com -Port 3306
# Expected: TcpTestSucceeded : False

# Linux/Mac:
nc -zv your-db.xxxx.region.rds.amazonaws.com 3306 -w 5
# Expected: Connection timed out

Good. The database is unreachable from the internet. That's what we want. Now we need a way IN. (You can also try other commands like telnet if you have those installed , these are most simplest ones. )

Phase 2: Launch the OpenVPN Server

Step 2.1: Create the VPN Security Group

EC2 Console → Security Groups → Create:

Name:  vpn-server-sg
VPC:   Your VPC

Inbound rules:

Port	Source	Why
1194/TCP	0.0.0.0/0	VPN clients connect from anywhere
22/TCP	YOUR-IP/32	SSH admin access, your IP only

"Wait, 0.0.0.0/0 on the VPN port? Isn't that what we're trying to avoid?"

Good instinct. But it's safe here because OpenVPN is protected by PKI certificates — without a valid signed certificate, you can't even start a handshake. Plus we'll add HMAC authentication (ta.key) that drops unsigned packets before any TLS processing. It's not like an open MySQL port where a username/password is your only defense.

SSH is restricted to your IP because SSH uses password/key auth — no certificate layer. Keep it locked down.

Step 2.2: Add VPN SG to the DB Security Group

Go back to private-db-sg and add:

MySQL/Aurora  3306  from vpn-server-sg  "Developer access via VPN"

This completes the security chain: VPN clients → OpenVPN EC2 (has vpn-server-sg) → NAT → RDS (accepts traffic from vpn-server-sg).

Step 2.3: Launch the EC2 Instance

EC2 Console → Launch instance:

Setting	Value	Why
AMI	Amazon Linux 2023 (ARM)	AWS-optimized, Graviton support
Type	t4g.micro	ARM = cheaper. Micro handles <20 VPN connections easily
Subnet	A public subnet	VPN server MUST be reachable from the internet
Auto-assign public IP	Enable	Needs internet reachability
Security group	`vpn-server-sg`	The one we just created
Storage	8 GB gp3	Minimal, sufficient

Step 2.4: Attach an Elastic IP

EC2 Console → Elastic IPs → Allocate → Associate with your instance.

Without an Elastic IP, the public IP changes every time you stop/start the instance. Every developer's VPN config has the server IP hardcoded — you don't want to redistribute configs every reboot.

Elastic IPs are free while attached to a running instance. They cost money if allocated but NOT attached (AWS charges for wasted IPs).

Step 2.5: SSH In

ssh -i "your-key.pem" ec2-user@YOUR-ELASTIC-IP

Amazon Linux uses ec2-user. Ubuntu uses ubuntu. Using the wrong username = "Permission denied."

Phase 3: Build the PKI (Your Own Certificate Authority)

This is where it gets interesting. We're building the same kind of certificate infrastructure that powers HTTPS across the entire internet — but for our VPN.

What is PKI and Why Should You Care?

Real-world analogy:

  Government (Certificate Authority)
       │
       ├── Issues passport to "VPN Server"
       │   └── Clients verify: "Is this the real server?"
       │
       ├── Issues passport to "Developer Tirth"
       │   └── Server verifies: "Is this an authorized developer?"
       │
       └── Revokes passport of "Developer Who Left"
           └── Their passport is now useless, even if they still have it

PKI gives us three superpowers:

Mutual authentication — server proves identity to client, client proves identity to server
Individual revocation — fire someone? Revoke their cert. Everyone else keeps working
No shared secrets — each developer has a unique key pair. Compromising one doesn't compromise others

Step 3.1: Install Packages

# Update system
sudo dnf update -y

# OpenVPN is in default Amazon Linux 2023 repos (no EPEL needed!)
sudo dnf install -y openvpn iptables-services

# EasyRSA is NOT in the repos — download from GitHub
cd ~
curl -LO https://github.com/OpenVPN/easy-rsa/releases/download/v3.2.2/EasyRSA-3.2.2.tgz
tar xzf EasyRSA-3.2.2.tgz
mv EasyRSA-3.2.2 openvpn-ca

The EPEL trap: Many guides tell you to install EPEL on Amazon Linux 2023. It fails with nothing provides redhat-release >= 9. Amazon Linux 2023 isn't RHEL 9 — it's Fedora-based but with its own package set. OpenVPN is already in the default repos. EasyRSA we grab manually.

What is EasyRSA? It's a wrapper around OpenSSL. OpenSSL can do everything EasyRSA does, but with commands like:

# Without EasyRSA (raw OpenSSL):
openssl req -new -keyout server.key -out server.req -nodes -subj "/CN=server"
openssl ca -in server.req -out server.crt -config openssl.cnf -batch

# With EasyRSA:
./easyrsa gen-req server nopass
./easyrsa sign-req server server

Same result. 10x simpler.

Step 3.2: Enable IP Forwarding

# Enable NOW (takes effect immediately)
sudo sysctl -w net.ipv4.ip_forward=1

# Make it survive reboots
echo 'net.ipv4.ip_forward = 1' | sudo tee /etc/sysctl.d/99-openvpn.conf
sudo sysctl --system

Why? By default, Linux drops packets that aren't addressed to itself. But VPN clients will send packets destined for the RDS (172.31.48.x), and they arrive at the VPN server (172.31.33.x). Without forwarding:

Without IP forwarding:
  Packet from 10.8.0.2 → destination 172.31.48.50
  Server: "That's not my IP. DROPPED."

With IP forwarding:
  Packet from 10.8.0.2 → destination 172.31.48.50
  Server: "Not mine, but I'll forward it." → routes to RDS

Step 3.3: Build the Certificate Authority

cd ~/openvpn-ca
./easyrsa init-pki
./easyrsa build-ca nopass
# When prompted for Common Name, enter something meaningful:
# "MyCompany VPN CA"

This creates two files that are the root of your entire security:

File	What	If compromised
`pki/ca.crt`	CA certificate (public)	Not a disaster — it's meant to be shared
`pki/private/ca.key`	CA private key	Game over. Attacker can forge any certificate. Rebuild everything.

Back up ca.key NOW. Store it encrypted somewhere safe. If you lose it, you can't sign new certs or revoke old ones. You'd rebuild the entire PKI from scratch.

Step 3.4: Generate Server Certificate

# Generate key + certificate signing request
./easyrsa gen-req vpn-server nopass

# Sign it with your CA (type "yes" when prompted)
./easyrsa sign-req server vpn-server

The server type in sign-req server embeds extendedKeyUsage = serverAuth in the certificate. This means a client certificate can't impersonate the server, and a server certificate can't be used as a client. Cryptographic separation of roles.

Step 3.5: Generate DH Parameters + TLS-Auth Key

# DH parameters (takes 1-5 minutes — it's finding large primes)
./easyrsa gen-dh

# TLS-Auth HMAC key
openvpn --genkey secret ta.key

DH (Diffie-Hellman): Allows client and server to agree on a shared encryption key without ever transmitting it. The math is beautiful — both sides compute the same secret independently, and an eavesdropper who sees the entire exchange still can't figure it out.

ta.key: An HMAC key that signs every single packet. If a packet arrives without a valid HMAC signature, it's dropped immediately — before any TLS processing, before any CPU-expensive crypto. This gives you:

DoS protection (attackers can't overwhelm TLS handshakes)
Port scan invisibility (port 1194 appears closed to scanners)

Step 3.6: Generate Your First Client Certificate

./easyrsa gen-req dev-yourname nopass
./easyrsa sign-req client dev-yourname

Per-developer certificates are non-negotiable. If a developer leaves:

With shared cert: revoke it → regenerate → redistribute to EVERYONE → downtime
With per-developer certs: ./easyrsa revoke dev-departed → done. Nobody else affected.

Step 3.7: Deploy Certs to OpenVPN Directory

sudo cp ~/openvpn-ca/pki/ca.crt /etc/openvpn/server/
sudo cp ~/openvpn-ca/pki/issued/vpn-server.crt /etc/openvpn/server/
sudo cp ~/openvpn-ca/pki/private/vpn-server.key /etc/openvpn/server/
sudo cp ~/openvpn-ca/pki/dh.pem /etc/openvpn/server/
sudo cp ~/openvpn-ca/ta.key /etc/openvpn/server/

# Verify the cert chain is valid:
openssl verify -CAfile /etc/openvpn/server/ca.crt /etc/openvpn/server/vpn-server.crt
# Expected: vpn-server.crt: OK

Phase 4: Configure OpenVPN + NAT

Step 4.1: Create Server Configuration

sudo tee /etc/openvpn/server/server.conf << 'EOF'
# ── Network ──
port 1194
proto tcp
dev tun

# ── Certificates ──
ca       /etc/openvpn/server/ca.crt
cert     /etc/openvpn/server/vpn-server.crt
key      /etc/openvpn/server/vpn-server.key
dh       /etc/openvpn/server/dh.pem
tls-auth /etc/openvpn/server/ta.key 0

# ── VPN Subnet ──
server 10.8.0.0 255.255.255.0

# ── Routing (THE important part) ──
push "route 172.31.0.0 255.255.0.0"
push "dhcp-option DNS 172.31.0.2"

# ── Security ──
cipher AES-256-GCM
auth SHA256

# ── Connection ──
keepalive 10 120
persist-key
persist-tun
ifconfig-pool-persist ipp.txt

# ── Logging ──
status      /var/log/openvpn-status.log
log-append  /var/log/openvpn.log
verb 3
EOF

Let me explain the three directives that matter most:

server 10.8.0.0 255.255.255.0 — Creates a virtual network for VPN clients. Server gets 10.8.0.1, first client gets 10.8.0.2, etc. This range is completely separate from your VPC (172.31.x.x) — that separation is intentional and important.

push "route 172.31.0.0 255.255.0.0" — This is the magic. When a client connects, OpenVPN adds a routing rule to their laptop: "send all 172.31.x.x traffic through the VPN tunnel." Without this push, the client's laptop would try to reach 172.31.48.50 through its ISP, which would fail (private IP, unroutable on the internet).

push "dhcp-option DNS 172.31.0.2" — Uses the VPC's built-in DNS resolver (always at VPC CIDR base + 2). This ensures the RDS hostname resolves to its private IP, not its (nonexistent) public one.

Step 4.2: Set Up NAT (The Networking Punchline)

Here's the problem: your VPN client has IP 10.8.0.2. The RDS is at 172.31.48.50. When the VPN server forwards the client's packet to the RDS, the RDS sees:

Packet arrives at RDS:
  Source: 10.8.0.2    ← "Who? I don't know this network."
  Dest:   172.31.48.50

RDS tries to respond to 10.8.0.2 → no route exists → DROPPED

The fix: MASQUERADE. The VPN server rewrites the source IP to its own VPC IP before forwarding:

After MASQUERADE:
  Source: 172.31.33.118  ← "That's a VPC IP! I know how to reach it."
  Dest:   172.31.48.50

RDS responds to 172.31.33.118 → VPN server → reverse-translates → client

Your home router does this exact same thing every day (192.168.x.x → your public IP). We're just doing it for VPN traffic.

# Find your network interface name
IFACE=$(ip -o -4 route show to default | awk '{print $5}')
echo "Interface: $IFACE"  # Probably "ens5" on AWS

# THE NAT RULE: rewrite source IP for VPN traffic going to VPC
sudo iptables -t nat -A POSTROUTING -s 10.8.0.0/24 -o $IFACE -j MASQUERADE

# FORWARD RULES: allow VPN traffic to pass through the server
sudo iptables -A FORWARD -s 10.8.0.0/24 -j ACCEPT
sudo iptables -A FORWARD -d 10.8.0.0/24 -j ACCEPT

# SAVE (or they vanish on reboot)
sudo service iptables save
sudo systemctl enable iptables
sudo systemctl start iptables

Breaking down the MASQUERADE rule:

sudo iptables -t nat -A POSTROUTING -s 10.8.0.0/24 -o ens5 -j MASQUERADE
                │        │              │               │         │
                │        │              │               │         └── Action: rewrite source IP
                │        │              │               └── Outgoing interface (to VPC)
                │        │              └── Match: source is VPN client
                │        └── Chain: after routing, before packet leaves
                └── Table: NAT (address translation)

Step 4.3: Start OpenVPN

sudo systemctl enable openvpn-server@server
sudo systemctl start openvpn-server@server

# Verify it's running
sudo systemctl status openvpn-server@server
# Look for: "active (running)"

# Verify the TUN interface exists
ip addr show tun0
# Should show: inet 10.8.0.1

What is openvpn-server@server? It's a systemd template. The part after @ is the config file name — it reads /etc/openvpn/server/server.conf. If you had another config called office.conf, you'd run openvpn-server@office.

What is tun0? A virtual network interface created by OpenVPN. Your server now has two interfaces:

ens5 — real network card, connects to VPC (172.31.33.x)
tun0 — virtual tunnel, connects to VPN network (10.8.0.1)

Encrypted VPN packets arrive on ens5:1194 → OpenVPN decrypts them → decrypted packets appear on tun0 → kernel routes them to the destination through ens5.

Phase 5: Connect and Test

Step 5.1: Create a Client Config Generator

mkdir -p ~/clients

cat > ~/generate-client.sh << 'SCRIPT'
#!/bin/bash
CLIENT=$1
ELASTIC_IP="YOUR-ELASTIC-IP-HERE"  # ← Replace with your Elastic IP

if [ -z "$CLIENT" ]; then
    echo "Usage: ./generate-client.sh <client-name>"
    exit 1
fi

cd ~/openvpn-ca

# Generate cert if it doesn't exist
if [ ! -f "pki/issued/${CLIENT}.crt" ]; then
    ./easyrsa gen-req ${CLIENT} nopass <<< ""
    echo "yes" | ./easyrsa sign-req client ${CLIENT}
fi

# Build self-contained .ovpn file
cat > ~/clients/${CLIENT}.ovpn << EOF
client
dev tun
proto tcp
remote ${ELASTIC_IP} 1194
resolv-retry infinite
nobind
persist-key
persist-tun
cipher AES-256-GCM
auth SHA256
key-direction 1
verb 3

<ca>
$(cat pki/ca.crt)
</ca>
<cert>
$(cat pki/issued/${CLIENT}.crt)
</cert>
<key>
$(cat pki/private/${CLIENT}.key)
</key>
<tls-auth>
$(cat ta.key)
</tls-auth>
EOF

echo "Created: ~/clients/${CLIENT}.ovpn"
SCRIPT

chmod +x ~/generate-client.sh

Why is everything embedded in one file? The .ovpn bundles the CA cert, client cert, client key, and TLS-auth key inline using XML-like tags. Instead of distributing 4 files + a config, the developer gets one file: import and connect.

Step 5.2: Generate and Download Your Config

# On the server:
./generate-client.sh dev-yourname

# On your laptop (download it):
scp -i "your-key.pem" ec2-user@YOUR-ELASTIC-IP:~/clients/dev-yourname.ovpn .

Security warning: This file contains private keys. Do NOT send it over Slack or email. Use SCP, S3 presigned URLs (with 1-hour expiry), or physical transfer.

Step 5.3: Connect

Install OpenVPN Connect (openvpn.net/client)
Import dev-yourname.ovpn
Click Connect

You should see: Connected, assigned IP 10.8.0.x.

Step 5.4: The Moment of Truth

With VPN connected:

mysql -h your-db.xxxx.region.rds.amazonaws.com -u admin -p
# Expected: Welcome to the MySQL monitor. 🎉

Disconnect VPN, try again:

mysql -h your-db.xxxx.region.rds.amazonaws.com -u admin -p --connect-timeout=10
# Expected: ERROR 2003 (HY000): Can't connect (Connection timed out)

That timeout IS the success. The database is invisible to the internet. Only VPN-connected users can reach it.

The Complete Packet Journey

Here's what happens end-to-end when you run mysql -h 172.31.48.50 through the VPN. Every single hop:

1. LAPTOP: "172.31.48.50? My routing table says → send through tun0 (VPN)"
     │
2. OPENVPN CLIENT: Encrypts packet + adds HMAC signature
     │
3. INTERNET: Encrypted blob travels to Elastic IP, port 1194
     │
4. VPN SERVER (ens5): Receives on port 1194
   → OpenVPN verifies HMAC (ta.key) → valid
   → Decrypts → pushes to tun0
     │
5. KERNEL: "Destination 172.31.48.50 isn't mine. IP forwarding is ON. Forward it."
     │
6. IPTABLES FORWARD: "-s 10.8.0.0/24 -j ACCEPT" → allowed
     │
7. IPTABLES NAT: MASQUERADE rewrites source 10.8.0.2 → 172.31.33.118
   (remembers the mapping for the return trip)
     │
8. VPC ROUTING: 172.31.48.50 → private subnet → arrives at RDS
     │
9. RDS SECURITY GROUP: "Is 172.31.33.118 in vpn-server-sg?" → YES → ALLOW
     │
───── RESPONSE ─────
     │
10. RDS responds to 172.31.33.118
    → iptables reverse NAT: → 10.8.0.2
    → kernel forwards to tun0
    → OpenVPN encrypts
    → sends back over internet
    → client decrypts
    → "Welcome to MySQL 8.0"

Ten hops. All invisible to the developer. They just type mysql -h ... and it works.

Operational Essentials

Adding a New Developer

# On the VPN server:
./generate-client.sh dev-newname
# Transfer the .ovpn file securely

Removing a Developer

cd ~/openvpn-ca

# Revoke their certificate
./easyrsa revoke dev-departed
# Type "yes"

# Regenerate the Certificate Revocation List
./easyrsa gen-crl

# Deploy it
sudo cp pki/crl.pem /etc/openvpn/server/

# Add to server.conf (first time only):
# crl-verify /etc/openvpn/server/crl.pem

# Restart
sudo systemctl restart openvpn-server@server

Their .ovpn file is now permanently useless. Even if they kept a copy, the server rejects revoked certificates.

Seeing Who's Connected

sudo cat /var/log/openvpn-status.log

Common Pitfalls (I Hit All of Them)

1. EPEL Doesn't Work on Amazon Linux 2023

Error: nothing provides redhat-release >= 9

Amazon Linux 2023 isn't RHEL 9. OpenVPN is in the default repos. Install with dnf install openvpn directly.

2. iptables Chain Names Are Case-Sensitive

sudo iptables -L forward -n -v
# ERROR: chain 'forward' in table 'filter' is incompatible

sudo iptables -L FORWARD -n -v
# Works! ✅

3. EasyRSA's `make-cadir` Doesn't Exist on Amazon Linux

Because we installed EasyRSA manually (not from a package). The directory was created by extracting the tarball. Just cd ~/openvpn-ca and run ./easyrsa directly.

4. Forgetting to Save iptables Rules

You add the perfect MASQUERADE rule. It works. You reboot the server. It's gone. Always:

sudo service iptables save
sudo systemctl enable iptables

5. Not Pushing DNS to Clients

Without push "dhcp-option DNS 172.31.0.2", the RDS hostname might resolve to the wrong IP (or not resolve at all) from the client's perspective. The VPC DNS resolver knows the private IPs.

What's Next: Production Cutover

Everything above creates a parallel test setup — new RDS, new subnets, new VPN. Your existing production database is untouched.

When you're ready to migrate the real database, only three things change:

Setting	Before	After
RDS subnet group	`default`	`my-private-db-subnet`
RDS public access	Yes	No
RDS security group	The old one with 0.0.0.0/0	`private-db-sg`

That's it. The VPN config, certificates, iptables rules, client .ovpn files — none of them change. They were designed to route ALL VPC traffic (172.31.0.0/16), not just one specific database. The production RDS lives within that range, so it's automatically covered.

The cutover causes ~5-15 minutes of database downtime while AWS reconfigures the network placement. Take a snapshot first. Have a rollback plan.

TL;DR

A subnet is "private" because its route table has no Internet Gateway route — that's it
OpenVPN + PKI gives you mutual certificate authentication, individual revocation, and no shared secrets
NAT/MASQUERADE lets VPN clients (10.8.0.x) talk to VPC resources (172.31.x.x) by rewriting source IPs
IP forwarding + iptables FORWARD rules are required for the server to route packets between networks
EasyRSA is a wrapper around OpenSSL — same crypto, 10x simpler commands
Amazon Linux 2023 has OpenVPN in default repos (no EPEL needed) but doesn't have EasyRSA (manual install)
Design your VPN to route the entire VPC CIDR — adding new private resources requires zero VPN changes

The database went from "accessible by the entire internet" to "accessible only through a certificate-authenticated encrypted tunnel." The infrastructure knowledge you build here — PKI, NAT, VPC networking, Linux packet routing — applies directly to Kubernetes, zero-trust architectures, and any serious infrastructure work.

This setup has been running in production since February 2026 on AWS (Amazon Linux 2023, OpenVPN 2.6.12, MySQL 8.0 on RDS, ap-south-1 region).

[Boost]

Tirth T — Fri, 20 Feb 2026 06:10:32 +0000

How I Tricked AWS Elastic Beanstalk Into Using pnpm

Tirth T ・ Feb 18

#beginners #devops #aws #npm

How I Tricked AWS Elastic Beanstalk Into Using pnpm

Tirth T — Wed, 18 Feb 2026 06:43:47 +0000

You've migrated your Node.js project to pnpm. Locally, everything works well — faster installs, cleaner node_modules, a strict lockfile.

Then you deploy to AWS Elastic Beanstalk.

And it fails.

Here's what happened when we pushed our first pnpm-based deployment:

[ERROR] An error occurred during execution of command [app-deploy] - 
[InstallDependencies]. Stop running the command.
Error: install dependencies fails: Command /bin/sh -c npm --omit=dev install 
failed with error exit status 1.
Stderr: npm warn old lockfile
npm warn old lockfile The package-lock.json file was created with an old version
npm error code EUSAGE
npm error
npm error `npm ci` can only install packages when your package.json and 
package-lock.json or npm-shrinkwrap.json are in sync. Please update your lock 
file with `npm install` before continuing.

The root cause: Elastic Beanstalk's Node.js platform hardcodes npm install during deployment. There's no config toggle, no environment variable, no checkbox in the console. It runs npm install. Period.

Our first fix attempt was to use corepack enable — the standard way to activate pnpm on a normal Node.js installation — inside a platform hook. That also failed:

.platform/hooks/prebuild/01_install_pnpm.sh: line 4: corepack: command not found
.platform/hooks/prebuild/01_install_pnpm.sh: line 22: pnpm: command not found

It turns out Amazon Linux 2023's Node.js package does NOT include corepack, unlike standard nodejs.org distributions. So not only does Beanstalk force npm install, you can't even install pnpm the standard way.

You can't change it. But you can trick it.

The Strategy: An npm Wrapper

Instead of fighting Beanstalk's lifecycle, we work with it. The approach:

Install pnpm on the instance (using Corepack or npm, depending on the platform)
Replace the npm binary with a wrapper script that intercepts install and ci commands and silently redirects them to pnpm
All other npm commands (--version, run, etc.) still use the original npm

Beanstalk thinks it's running npm. It's actually running pnpm. It never knows the difference.

Here's the architecture:

Beanstalk Lifecycle
    │
    ├─ prebuild hook ──► Install pnpm via Corepack
    │                    Replace /usr/bin/npm with wrapper
    │
    ├─ npm install ────► Wrapper intercepts → pnpm install
    │
    └─ npm start ──────► Wrapper passes through → original npm

Project Structure

Elastic Beanstalk uses a .platform directory for lifecycle hooks. Here's what we need:

your-project/
├── .platform/
│   └── hooks/
│       └── prebuild/
│           └── 01_install_pnpm.sh    ← The core script
├── Procfile                           ← web: pnpm start
├── package.json
├── pnpm-lock.yaml                     ← Your pnpm lockfile
└── ...

Why hooks/prebuild?

Beanstalk runs hooks in this order: prebuild → npm install → predeploy → postdeploy. We need pnpm ready before Beanstalk attempts to install dependencies.

The Bug That Almost Broke Everything

Our first version of the script had a critical chicken-and-egg bug:

1. Script runs: npm install -g pnpm
2. Script replaces /usr/bin/npm with wrapper that calls pnpm
3. Deployment FAILS (some other reason)
4. Next deployment tries again...
5. npm is now broken! It tries to call pnpm, but pnpm doesn't exist!
6. "pnpm: command not found" — forever

The wrapper was created before verifying pnpm was properly installed. If anything failed between installation and wrapper creation, npm would be permanently broken on the instance — it would try to call pnpm (which doesn't exist), and since npm is broken, you can't even npm install -g pnpm to fix it.

This is why the script must:

Always restore the original npm first (safety net)
Verify pnpm is installed before creating the wrapper
Only then replace npm with the wrapper

The Platform Hook Script

Create .platform/hooks/prebuild/01_install_pnpm.sh:

#!/usr/bin/env bash
set -e

echo "=== Starting pnpm installation via Corepack ==="

# ─── SAFETY NET ─────────────────────────────────────
# If a previous deployment failed mid-script, the original
# npm binary might be stuck at /usr/bin/npm_original.
# Restore it before we do anything else.
if [ -f /usr/bin/npm_original ]; then
    echo "Restoring original npm from backup..."
    mv /usr/bin/npm_original /usr/bin/npm
fi

# ─── INSTALL COREPACK ───────────────────────────────
# Amazon Linux 2023 does NOT include corepack unlike
# standard nodejs.org Node.js distributions.
# We must bootstrap it using npm first.
echo "Installing corepack..."
npm install -g corepack

# ─── ENABLE PNPM VIA COREPACK ──────────────────────
# 'corepack enable' creates a shim (lightweight proxy) at /usr/bin/pnpm
# It doesn't download pnpm yet — just creates the pointer.
# 'corepack prepare' actually downloads the binary and caches it,
# preventing the "Do you want to download?" prompt during automated builds.
echo "Enabling corepack and installing pnpm..."
corepack enable
corepack prepare pnpm@latest --activate

# ─── VERIFY INSTALLATION ───────────────────────────
if ! command -v pnpm &> /dev/null; then
    echo "ERROR: pnpm installation failed!"
    exit 1
fi
echo "pnpm installed successfully: $(pnpm --version)"

# ─── THE NPM WRAPPER ───────────────────────────────
# Only AFTER pnpm is verified, it's safe to create the wrapper.
# We backup the real npm binary, then replace it with a script
# that intercepts install/ci commands and redirects to pnpm.
echo "Creating npm wrapper..."
mv /usr/bin/npm /usr/bin/npm_original

cat > /usr/bin/npm << 'EOF'
#!/bin/bash

# Loop through ALL arguments — not just $1.
# Beanstalk runs: npm --omit=dev install
# If you only check $1, you'd see --omit=dev and miss 'install' entirely.
for arg in "$@"; do
    case "$arg" in
        install|ci|add)
            echo "[npm-wrapper] Intercepted install → using pnpm"
            pnpm install --prod --frozen-lockfile --ignore-scripts
            exit $?
            ;;
        rebuild|update|prune|dedupe)
            echo "[npm-wrapper] Intercepted $arg → skipping (pnpm handles this)"
            exit 0
            ;;
    esac
done

# Everything else (--version, run, start, etc) → use original npm
/usr/bin/npm_original "$@"
EOF

chmod +x /usr/bin/npm

echo "=== pnpm installation complete ==="

Important: The file MUST have Linux line endings (LF, not CRLF). If you're on Windows, configure your editor or use git config core.autocrlf input.

Make It Executable

This is the step everyone forgets. The hook script must be executable, or Beanstalk silently ignores it:

# If you're on macOS/Linux
chmod +x .platform/hooks/prebuild/01_install_pnpm.sh

# If you're on Windows, use Git to track the permission
git update-index --chmod=+x .platform/hooks/prebuild/01_install_pnpm.sh

Understanding the Wrapper: The `$1` Bug

Our original wrapper only checked $1:

# BROKEN — only checks the first argument
if [ "$1" = "install" ] || [ "$1" = "ci" ]; then
    pnpm install --prod --frozen-lockfile --ignore-scripts
else
    /usr/bin/npm_original "$@"
fi

This broke because Beanstalk runs:

npm --omit=dev install

Here, $1 is --omit=dev, not install. The wrapper missed it entirely and fell through to the original npm, which failed because there's no package-lock.json.

The fix was to loop through ALL arguments using for arg in "$@" and use a case statement:

for arg in "$@"; do
    case "$arg" in
        install|ci|add)
            pnpm install --prod --frozen-lockfile --ignore-scripts
            exit $?
            ;;
        rebuild|update|prune|dedupe)
            exit 0  # Skip — pnpm handles it
            ;;
    esac
done

Why intercept `rebuild`, `update`, `prune`, and `dedupe`?

Beanstalk runs npm rebuild and sometimes npm prune after install. With pnpm, these are unnecessary — pnpm handles everything during install. We skip them with exit 0 so Beanstalk doesn't interpret it as a failure.

Why `--prod --frozen-lockfile --ignore-scripts`?

Flag	Why
`--prod`	Only install production dependencies (skip devDependencies)
`--frozen-lockfile`	Fail if lockfile is out of sync — prevents "works on my machine" bugs
`--ignore-scripts`	Skip postinstall scripts during deploy for security and speed

Understanding Corepack Shims

When you run corepack enable, it doesn't install the real pnpm binary. It creates a shim — a lightweight proxy script at /usr/bin/pnpm that intercepts your command.

Running corepack prepare pnpm@latest --activate actually pre-downloads the real binary into the system cache. Without this step, the shim would prompt:

! Corepack is about to download https://registry.npmjs.org/pnpm/-/pnpm-10.28.2.tgz
? Do you want to continue? [Y/n]

This prompt would hang an automated deployment. The --activate flag ensures it downloads silently.

Note: If you SSH into the instance as a different user (e.g., ec2-user), you may still see this prompt because the user-specific cache is empty. The web application runs fine because the deployment user (root/webapp) already had pnpm "prepared" by the script.

Deploy and Verify

Push your code and watch the Beanstalk logs. Here's what a successful deployment looks like (from our actual eb-hooks.log):

=== Starting pnpm installation via Corepack ===
Restoring original npm from backup...
Installing corepack...
changed 1 package in 1s
Enabling corepack and installing pnpm...
Preparing pnpm@latest for immediate activation...
pnpm installed successfully: 10.28.2
Creating npm wrapper...
=== pnpm installation complete ===

Performance Results (February 2, 2026)

These are actual numbers from our production Elastic Beanstalk environment after fixing the npm wrapper bug:

Metric	Before (npm)	After (pnpm)	Improvement
Total Deploy Stage	~6 minutes	1 min 6 sec	5.5x faster
Package Install	~3 minutes	13 seconds	14x faster
npm rebuild	~5 seconds	Skipped	N/A

Troubleshooting

Key Log Files

Log File	Purpose
`/var/log/eb-hooks.log`	Prebuild script output (your script logs here)
`/var/log/eb-engine.log`	Deployment steps, artifact downloads
`/var/log/web.stdout.log`	Application runtime output/errors

# View last 100 lines of each log
sudo tail -100 /var/log/eb-hooks.log
sudo tail -100 /var/log/eb-engine.log
sudo tail -100 /var/log/web.stdout.log

Diagnosing the npm Wrapper State

# Check if npm is original or wrapper
cat /usr/bin/npm

# If it shows the wrapper script, npm has been replaced
# If it shows the original npm content (long script), it's untouched

# Check if backup exists
ls -la /usr/bin/npm_original

# Check if pnpm exists
which pnpm
pnpm --version

Fixing Broken State Manually

If npm is broken on the instance (wrapper exists but pnpm doesn't):

# Restore the original npm
sudo mv /usr/bin/npm_original /usr/bin/npm

# Then install pnpm manually
sudo npm install -g pnpm

# Verify
pnpm --version

Running the App Manually via SSH

SSH sessions don't have Elastic Beanstalk environment variables. Load them first:

sudo su -
source /opt/elasticbeanstalk/deployment/env
cd /var/app/current
pnpm start

Common Errors & Solutions

Error	Cause	Solution
`corepack: command not found`	Amazon Linux doesn't include corepack	Use `npm install -g corepack` first
`pnpm: command not found` (inside npm)	npm wrapper created before pnpm installed	Restore `npm_original`, then reinstall
App crashes with missing env var	SSH session doesn't have EB env vars	Use `source /opt/elasticbeanstalk/deployment/env`
Rollback loop	Old broken version keeps deploying	Disable auto-rollback in CodePipeline (set to "None")

TL;DR

AWS Elastic Beanstalk hardcodes npm install — you can't change it
Amazon Linux 2023 doesn't include corepack — you must bootstrap it with npm install -g corepack
Create a .platform/hooks/prebuild/ hook that installs pnpm and replaces npm with a wrapper
The wrapper must loop through ALL arguments (not just $1) because Beanstalk runs npm --omit=dev install
Always restore the original npm first to avoid the chicken-and-egg bug
Result: package install dropped from ~3 minutes to 13 seconds

Should AWS Just Support pnpm Natively?

Yes. But until that day comes, this wrapper approach is production-tested and running across multiple services without issues.

If you found this useful, feel free to share or leave a comment.

This approach has been running in production since December 2025 across multiple Node.js services on Elastic Beanstalk (Amazon Linux 2023, Node.js 22.x platform).

Forem: Tirth T

[Boost]

I Moved My Database Behind a VPN on AWS — Here's Every Step (With the Networking Concepts That Actually Matter)

Tirth T ・ Mar 12

I Moved My Database Behind a VPN on AWS — Here's Every Step (With the Networking Concepts That Actually Matter)

The Moment I Realized Our Database Was Naked

What We're Building

Why OpenVPN? (And Not Tailscale or WireGuard)

Phase 1: Build the Private Network Layer

The One Thing That Makes a Subnet Private

Step 1.1: Create Two Private Subnets

Step 1.2: Create the Private Route Table

Step 1.3: Associate Your Subnets

Step 1.4: Create the RDS Subnet Group

Step 1.5: Launch Your RDS in the Private Subnet

Step 1.6: Configure the DB Security Group

Verify: Can You Reach the Database?

Phase 2: Launch the OpenVPN Server

Step 2.1: Create the VPN Security Group

Step 2.2: Add VPN SG to the DB Security Group

Step 2.3: Launch the EC2 Instance

Step 2.4: Attach an Elastic IP

Step 2.5: SSH In

Phase 3: Build the PKI (Your Own Certificate Authority)

What is PKI and Why Should You Care?

Step 3.1: Install Packages

Step 3.2: Enable IP Forwarding

Step 3.3: Build the Certificate Authority

Step 3.4: Generate Server Certificate

Step 3.5: Generate DH Parameters + TLS-Auth Key

Step 3.6: Generate Your First Client Certificate

Step 3.7: Deploy Certs to OpenVPN Directory

Phase 4: Configure OpenVPN + NAT

Step 4.1: Create Server Configuration

Step 4.2: Set Up NAT (The Networking Punchline)

Step 4.3: Start OpenVPN

Phase 5: Connect and Test

Step 5.1: Create a Client Config Generator

Step 5.2: Generate and Download Your Config

Step 5.3: Connect

Step 5.4: The Moment of Truth

The Complete Packet Journey

Operational Essentials

Adding a New Developer

Removing a Developer

Seeing Who's Connected

Common Pitfalls (I Hit All of Them)

1. EPEL Doesn't Work on Amazon Linux 2023

2. iptables Chain Names Are Case-Sensitive

3. EasyRSA's make-cadir Doesn't Exist on Amazon Linux

4. Forgetting to Save iptables Rules

5. Not Pushing DNS to Clients

What's Next: Production Cutover

TL;DR

[Boost]

How I Tricked AWS Elastic Beanstalk Into Using pnpm

Tirth T ・ Feb 18

How I Tricked AWS Elastic Beanstalk Into Using pnpm

The Strategy: An npm Wrapper

Project Structure

The Bug That Almost Broke Everything

The Platform Hook Script

Make It Executable

Understanding the Wrapper: The $1 Bug

Why intercept rebuild, update, prune, and dedupe?

Why --prod --frozen-lockfile --ignore-scripts?

Understanding Corepack Shims

Deploy and Verify

Performance Results (February 2, 2026)

Troubleshooting

Key Log Files

Diagnosing the npm Wrapper State

Fixing Broken State Manually

Running the App Manually via SSH

Common Errors & Solutions

TL;DR

Should AWS Just Support pnpm Natively?

3. EasyRSA's `make-cadir` Doesn't Exist on Amazon Linux

Understanding the Wrapper: The `$1` Bug

Why intercept `rebuild`, `update`, `prune`, and `dedupe`?

Why `--prod --frozen-lockfile --ignore-scripts`?