Forem: Lucas

Building Hybrid Search for RAG: Combining pgvector and Full-Text Search with Reciprocal Rank Fusion

Lucas — Thu, 12 Feb 2026 04:10:57 +0000

Most RAG tutorials show you the happy path: embed your documents, store them in a vector database, do a similarity search, done. Ship it.

Then you put it in front of real users and discover that semantic search alone misses obvious keyword matches, exact product names get hallucinated into approximate nonsense, and your retrieval quality sits at maybe 60%.

This post covers how we fixed it — using nothing but PostgreSQL. pgvector for semantic search, tsvector for full-text search, and Reciprocal Rank Fusion to merge the results. No Pinecone. No Elasticsearch. Just Postgres.

TL;DR — Pure vector search got us ~62% retrieval precision. Adding full-text search + RRF fusion bumped it to ~84%, with near-perfect exact-match queries. Zero additional infrastructure — it's all just SQL.

Why Not Just Use Vector Search?

Vector (semantic) search is great at understanding meaning. If a user asks "how do I reduce cloud costs?" it will find documents about "infrastructure cost optimization" even if those exact words don't appear.

But it has blind spots. Here's how the two approaches compare:

Query type	Vector search	Full-text search
"infrastructure cost optimization"	Finds it even if phrased differently	Misses unless exact words match
"PCI-DSS v4" (exact standard)	Returns generic "security compliance" docs	Nails it
"ECS" (acronym)	Matches "container services" broadly	Finds exact AWS ECS references
"SQS DLQ config" (precise query)	Diluted by semantically similar noise	Direct hit
"how do I handle errors gracefully?"	Understands intent, finds relevant patterns	Misses if docs use different wording

Neither approach is sufficient on its own. The answer is to use both.

Architecture Overview

Both searches run in parallel against the same PostgreSQL database. RRF merges the ranked lists into a single result set.

Setting Up the Database

pgvector for Semantic Search

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE document_chunks (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    document_id UUID NOT NULL REFERENCES documents(id),
    content TEXT NOT NULL,
    embedding vector(3072),  -- text-embedding-3-large dimensions
    metadata JSONB DEFAULT '{}',
    created_at TIMESTAMPTZ DEFAULT now()
);

-- HNSW index for approximate nearest neighbor search
CREATE INDEX idx_chunks_embedding ON document_chunks
    USING hnsw (embedding vector_cosine_ops)
    WITH (m = 16, ef_construction = 128);

Why HNSW over IVFFlat? HNSW gives better recall at query time without needing to tune nprobe. IVFFlat is faster to build but requires choosing the number of lists upfront and tuning probe count per query. For production RAG where retrieval quality matters more than index build time, HNSW is the better default.

tsvector for Full-Text Search

ALTER TABLE document_chunks
    ADD COLUMN search_vector tsvector
    GENERATED ALWAYS AS (to_tsvector('english', content)) STORED;

CREATE INDEX idx_chunks_search ON document_chunks USING gin(search_vector);

Using a GENERATED ALWAYS column means the search vector stays in sync with content automatically — no triggers, no application logic to maintain.

The Search Queries

Semantic Search

async def semantic_search(
    session: AsyncSession,
    query_embedding: list[float],
    workspace_id: uuid.UUID,
    limit: int = 20,
) -> list[SearchResult]:
    """Find chunks by vector similarity using cosine distance."""
    stmt = (
        select(
            DocumentChunk.id,
            DocumentChunk.content,
            DocumentChunk.document_id,
            (1 - DocumentChunk.embedding.cosine_distance(query_embedding)).label(
                "similarity"
            ),
        )
        .join(Document, DocumentChunk.document_id == Document.id)
        .where(Document.workspace_id == workspace_id)
        .order_by(
            DocumentChunk.embedding.cosine_distance(query_embedding)
        )
        .limit(limit)
    )

    results = await session.execute(stmt)
    return [
        SearchResult(
            chunk_id=row.id,
            content=row.content,
            document_id=row.document_id,
            score=float(row.similarity),
        )
        for row in results
    ]

Full-Text Search

async def fulltext_search(
    session: AsyncSession,
    query: str,
    workspace_id: uuid.UUID,
    limit: int = 20,
) -> list[SearchResult]:
    """Find chunks by full-text relevance using ts_rank."""
    ts_query = func.plainto_tsquery("english", query)

    stmt = (
        select(
            DocumentChunk.id,
            DocumentChunk.content,
            DocumentChunk.document_id,
            func.ts_rank(DocumentChunk.search_vector, ts_query).label("relevance"),
        )
        .join(Document, DocumentChunk.document_id == Document.id)
        .where(
            Document.workspace_id == workspace_id,
            DocumentChunk.search_vector.op("@@")(ts_query),
        )
        .order_by(func.ts_rank(DocumentChunk.search_vector, ts_query).desc())
        .limit(limit)
    )

    results = await session.execute(stmt)
    return [
        SearchResult(
            chunk_id=row.id,
            content=row.content,
            document_id=row.document_id,
            score=float(row.relevance),
        )
        for row in results
    ]

Reciprocal Rank Fusion

Here's where the magic happens. RRF merges two ranked lists without needing to normalize their scores — which is important because cosine similarity scores and ts_rank scores are on completely different scales.

The formula is simple:

RRF_score(d) = Σ  1 / (k + rank_i(d))

Where k is a constant (typically 60) and rank_i(d) is the rank of document d in the i-th result list.

Why not just normalize and average the scores? Cosine similarity and ts_rank have fundamentally different distributions. Normalizing them into a common range introduces arbitrary decisions about scaling. RRF side-steps this entirely by only caring about rank position, not raw scores. A document ranked #1 in both lists will always beat a document ranked #5 in one and #1 in another — regardless of what the underlying scores looked like.

def reciprocal_rank_fusion(
    result_lists: list[list[SearchResult]],
    k: int = 60,
) -> list[SearchResult]:
    """
    Merge multiple ranked result lists using Reciprocal Rank Fusion.

    The k parameter controls how much we trust top-ranked results.
    Higher k = more equal weighting. Default 60 is from the original
    Cormack et al. (2009) paper.
    """
    rrf_scores: dict[uuid.UUID, float] = {}
    results_by_id: dict[uuid.UUID, SearchResult] = {}

    for result_list in result_lists:
        for rank, result in enumerate(result_list, start=1):
            rrf_scores[result.chunk_id] = rrf_scores.get(
                result.chunk_id, 0.0
            ) + 1.0 / (k + rank)
            results_by_id[result.chunk_id] = result

    sorted_ids = sorted(rrf_scores, key=rrf_scores.get, reverse=True)

    return [
        SearchResult(
            chunk_id=chunk_id,
            content=results_by_id[chunk_id].content,
            document_id=results_by_id[chunk_id].document_id,
            score=rrf_scores[chunk_id],
        )
        for chunk_id in sorted_ids
    ]

Putting It Together

async def hybrid_search(
    session: AsyncSession,
    query: str,
    query_embedding: list[float],
    workspace_id: uuid.UUID,
    limit: int = 10,
) -> list[SearchResult]:
    """
    Run semantic and full-text search in parallel,
    then merge results with RRF.
    """
    semantic_results, fulltext_results = await asyncio.gather(
        semantic_search(session, query_embedding, workspace_id, limit=20),
        fulltext_search(session, query, workspace_id, limit=20),
    )

    fused = reciprocal_rank_fusion([semantic_results, fulltext_results])

    return fused[:limit]

Both searches run concurrently with asyncio.gather. We fetch 20 candidates from each, fuse them, and return the top 10. Over-fetching from each source gives RRF more signal to work with.

Why Not a Dedicated Vector Database?

We considered Pinecone, Weaviate, and Qdrant. We stuck with PostgreSQL + pgvector for several reasons:

Concern	Dedicated vector DB	PostgreSQL + pgvector
Infrastructure	Separate service to monitor, back up, patch, pay for	Already running Aurora PostgreSQL
Consistency	Eventual consistency between stores	Single transaction — deletes are atomic
Full-text search	Need a third system (Elasticsearch, etc.)	`tsvector` is built in
Compliance (SOC2)	Every new service adds audit scope	One set of encryption, access controls, audit trails
Performance at our scale	Faster at 10M+ vectors	<50ms HNSW search at ~500K chunks

When would we switch? If we hit tens of millions of vectors and needed sub-10ms p99 latency, a dedicated solution would make sense. At ~500K chunks, pgvector is more than fast enough.

Production Lessons

Here's what we learned running this in production:

1. Over-fetch, then trim.
Pulling 20 results from each source and fusing to 10 consistently outperformed pulling 10 from each. More candidates = better fusion signal.

2. Full-text search needs query preprocessing.
plainto_tsquery handles most cases, but for advanced users we added websearch_to_tsquery support which handles quoted phrases and boolean operators.

3. Monitor both sides independently.
We track semantic search latency and full-text search latency separately. If pgvector's HNSW index needs a rebuild, we see it in the semantic search p99 without it being masked by fast full-text results.

4. Embedding model upgrades require re-indexing.
When we considered switching embedding models, we realized every chunk needs re-embedding. We built this into our pipeline as a background job with progress tracking. Plan for this from day one.

5. RRF's k=60 is a reasonable default, but test it.
We ran evaluations with k values from 10 to 100. For our dataset, 60 was optimal. Your mileage will vary — the important thing is to have an evaluation set so you can actually measure.

Results

After switching from pure semantic search to hybrid search with RRF:

Metric	Before (vector only)	After (hybrid + RRF)
Retrieval precision	~62%	~84%
Exact-match queries (product names, error codes)	Unreliable	Near-perfect
Additional infrastructure	—	None. It's all just SQL.

Hybrid search isn't novel research — the pattern has been around for years in traditional information retrieval. But in the RAG gold rush, most tutorials skip it entirely in favor of pure vector search. If your RAG pipeline is struggling with retrieval quality, this is likely the highest-ROI improvement you can make.

[Code examples in this post are illustrative patterns based on production experience, not copy-paste from a proprietary codebase.]

Building a SOC2-Compliant Azure Multi-Subscription Architecture with Terraform

Lucas — Wed, 17 Dec 2025 08:47:54 +0000

A deep dive into implementing Microsoft's Cloud Adoption Framework Landing Zones with Terraform, and why Azure's approach to multi-tenancy requires a fundamentally different mindset than AWS.

The Challenge: "Just Make It Work Like AWS"

When our team at Foo was tasked with building a SOC2-compliant infrastructure on Azure, the initial instinct was simple: "Let's just replicate our AWS multi-account strategy." After all, we already had a battle-tested AWS organization with:

9+ AWS accounts (dev, staging, prod, security-tooling, log-archive, shared-services, etc.)
Service Control Policies (SCPs) for guardrails
IAM Identity Center (SSO) for centralized access
AWS Organizations for hierarchy

Spoiler alert: This approach would have been a costly mistake on Azure.

Here's why, and what we built instead.

🚨 The Azure Subscription Limit Reality Check

Unlike AWS, where you can spin up accounts almost endlessly (soft limit of ~10, easily increased to hundreds), Azure has hard limits on subscriptions that require engagement with Microsoft support:

Cloud Provider	Isolation Unit	Typical Count	Limit Increase
AWS	Account	10-100+	Easy (support ticket)
Azure	Subscription	3-10	Requires Microsoft engagement

This isn't just a technical limitation—it's a fundamental architectural constraint that shapes how you design your Azure landing zone.

💡 Key Insight: Azure's answer to "we need more isolation" isn't "create more subscriptions." It's Resource Groups with RBAC.

The Architecture: Microsoft CAF Landing Zone Pattern

After evaluating several approaches, we landed on Microsoft's Cloud Adoption Framework (CAF) Landing Zone architecture. Here's our 4-tier Management Group hierarchy:

5 Subscriptions vs 9+ AWS Accounts

Here's how we consolidated:

AWS Account	Azure Equivalent	Strategy
Dev Account	Corp-NonProduction Sub	Use Resource Groups: `rg-foo-preview-*`
Staging Account	Corp-NonProduction Sub	Use Resource Groups: `rg-foo-staging-*`
Prod Account	Corp-Production Sub	Dedicated subscription
Security-Tooling	Management Sub	Consolidated with monitoring
Log-Archive	Management Sub	Storage accounts with lifecycle policies
Shared-Services	Connectivity Sub	Hub networking
Network Account	Connectivity Sub	Azure Firewall, VPN, DNS
Monitoring	Management Sub	Log Analytics, Azure Monitor
Sandbox	Sandbox Sub	Developer experimentation

Result: 5 subscriptions doing the work of 9 AWS accounts.

Show Me The Code: Terraform Implementation

Management Group Hierarchy

Here's how we define the 4-tier CAF hierarchy in Terraform:

# management_groups.tf
# Azure Management Groups - Microsoft CAF Landing Zone Architecture

# Tier 1: Root Management Group
resource "azurerm_management_group" "foo" {
  display_name = var.organization_name
  name         = var.organization_name

  timeouts {
    create = "30m"  # MGs can take a while!
    delete = "30m"
  }
}

# Tier 2: Category Management Groups (Platform, Landing Zones, Sandbox)
resource "azurerm_management_group" "tier2_groups" {
  for_each = {
    for k, v in var.management_groups : k => v
    if v.parent_id == "foo"
  }

  display_name               = each.key
  name                       = each.key
  parent_management_group_id = azurerm_management_group.foo.id

  depends_on = [azurerm_management_group.foo]
}

# Tier 3: Specialization (Connectivity, Management, Corp)
resource "azurerm_management_group" "tier3_groups" {
  for_each = {
    for k, v in var.management_groups : k => v
    if v.parent_id == "Platform" || v.parent_id == "LandingZones"
  }

  display_name = each.key
  name         = each.key
  parent_management_group_id = lookup(
    { for k, v in azurerm_management_group.tier2_groups : k => v.id },
    each.value.parent_id,
    azurerm_management_group.foo.id
  )
}

# Tier 4: Environment (Prod, Non-Prod)
resource "azurerm_management_group" "tier4_groups" {
  for_each = {
    for k, v in var.management_groups : k => v
    if v.parent_id == "Corp"
  }

  display_name = each.key
  name         = each.key
  parent_management_group_id = azurerm_management_group.tier3_groups["Corp"].id
}

Subscription to Management Group Association

The key to this architecture is associating existing subscriptions with the hierarchy:

# subscriptions.tf
locals {
  subscription_to_mg_mapping = {
    "management"   = "Management"
    "corp-nonprod" = "NonProd"
    "corp-prod"    = "Prod"
    "connectivity" = "Connectivity"
    "sandbox"      = "Sandbox"
  }
}

resource "azurerm_management_group_subscription_association" "assignments" {
  for_each = local.subscription_to_mg_mapping

  management_group_id = lookup(local.all_management_group_ids, each.value)
  subscription_id     = "/subscriptions/${var.subscription_ids[each.key]}"
}

SOC2 Compliance: Azure Policy as Your Guardrails

For SOC2 Type II compliance, we implemented 7 custom Azure Policies that enforce security controls across the entire organization:

Legend: 🔴 Deny policies (blocking) | 🟢 Audit policies (non-blocking)

Policy Definition Example: Baseline Security

Here's a real policy that enforces SOC2 CC6.6 (Encryption) and CC7.2 (Monitoring):

# modules/policy/main.tf
resource "azurerm_policy_definition" "baseline_security" {
  name         = "foo-soc2-baseline-security"
  policy_type  = "Custom"
  mode         = "All"
  display_name = "SOC2 Baseline Security Controls"
  description  = "Baseline security - prevents deletion of critical resources"

  metadata = jsonencode({
    category = "Security"
    version  = "1.0.0"
    SOC2     = "CC6.6, CC7.2, CC8.1"
  })

  management_group_id = var.management_group_id

  policy_rule = file("${path.module}/policies/baseline_security.json")
}

And the policy rule itself (policies/baseline_security.json):

{
  "if": {
    "anyOf": [
      {
        "allOf": [
          { "field": "type", "equals": "Microsoft.Insights/activityLogAlerts" },
          { "field": "Microsoft.Insights/activityLogAlerts/enabled", "equals": "false" }
        ]
      },
      {
        "allOf": [
          { "field": "type", "equals": "Microsoft.OperationalInsights/workspaces" },
          { "field": "Microsoft.OperationalInsights/workspaces/retentionInDays", "less": 90 }
        ]
      },
      {
        "allOf": [
          { "field": "type", "equals": "Microsoft.Storage/storageAccounts" },
          { "field": "Microsoft.Storage/storageAccounts/encryption.requireInfrastructureEncryption", "notEquals": "true" }
        ]
      }
    ]
  },
  "then": {
    "effect": "deny"
  }
}

This policy blocks:

Disabling activity log alerts
Creating Log Analytics workspaces with < 90 days retention
Creating storage accounts without infrastructure encryption

Policy Assignment at Management Group Level

resource "azurerm_management_group_policy_assignment" "baseline_security" {
  name                 = "soc2-baseline-security"
  display_name         = "SOC2 Baseline Security Controls"
  management_group_id  = var.management_group_id
  policy_definition_id = azurerm_policy_definition.baseline_security.id

  metadata = jsonencode({
    Organization = "Foo"
    ManagedBy    = "Terraform"
    Purpose      = "SOC2 Compliance Guardrails"
  })
}

RBAC: The Azure AD Groups Strategy

Instead of AWS SSO Permission Sets, we use Azure AD Groups with RBAC role assignments:

# modules/iam/groups.tf
resource "azuread_group" "groups" {
  for_each = var.groups

  display_name     = each.value.display_name
  description      = each.value.description
  security_enabled = true

  prevent_duplicate_names = true
}

# modules/iam/role_assignments.tf
resource "azurerm_role_assignment" "group_assignments" {
  for_each = local.all_role_assignments

  scope                = each.value.scope
  role_definition_name = each.value.role_definition_name
  principal_id         = azuread_group.groups[each.value.group_key].object_id

  description = "Managed by Terraform - Foo IAM Module"
}

Our 7 Default Groups

Group	Platform Subs	Workload Subs	Purpose
Platform-Team	Contributor	Reader	Platform engineers
Security-Team	Security Admin	Security Reader	Security engineers
BreakGlass-Admins	Owner (at root)	-	Emergency access
Finance-Team	Cost Mgmt Reader	Cost Mgmt Reader	Billing/finance
ReadOnly-Users	Reader	Reader	Auditors
Dev-Team	-	Contributor (nonprod)	Developers
DevOps-Team	Reader	Contributor	DevOps engineers

The Resource Group Pattern: Your New Best Friend

Here's where Azure truly differs from AWS. Instead of creating separate subscriptions for staging vs preview, we use Resource Groups:

Corp-NonProduction Subscription
├── rg-foo-staging-api
├── rg-foo-staging-database
├── rg-foo-staging-network
├── rg-foo-preview-api
├── rg-foo-preview-database
└── rg-foo-preview-network

Each Resource Group gets its own RBAC assignments:

# Staging RGs: Only staging team
resource "azurerm_role_assignment" "staging_contributor" {
  scope                = azurerm_resource_group.staging_api.id
  role_definition_name = "Contributor"
  principal_id         = azuread_group.staging_team.object_id
}

# Preview RGs: All developers
resource "azurerm_role_assignment" "preview_contributor" {
  scope                = azurerm_resource_group.preview_api.id
  role_definition_name = "Contributor"
  principal_id         = azuread_group.dev_team.object_id
}

SOC2 Control Mapping

Here's how our implementation maps to SOC2 Trust Services Criteria:

SOC2 Control	Azure Implementation
CC6.1 - Logical Access	Azure AD groups, MFA via Conditional Access, PIM for JIT access
CC6.6 - Encryption	Azure Policy enforcing encryption at rest, TLS 1.2, Key Vault
CC7.2 - Monitoring	Log Analytics (90-day retention), Activity Logs, Azure Monitor
CC7.3 - Incident Response	Azure Sentinel (SIEM), Alert Rules, Action Groups
CC8.1 - Change Management	Activity Logs, Azure Policy audit, Git-based IaC
CC9.2 - Risk Mitigation	Region restrictions, Defender for Cloud, Budget alerts

Cost Impact: The Numbers

Running our 5-subscription architecture vs a 9-account AWS-style approach:

Metric	Azure CAF (5 subs)	AWS-style (9+ subs)	Savings
Management overhead	Low	High	~50% less ops time
Subscription costs	$0	$0	Same
Log Analytics	1 central workspace	9 workspaces	~60% cost reduction
Policy management	1 root assignment	9 assignments	Simpler

Lessons Learned

1. Management Groups Take Time

Azure Management Groups can take 10-15 minutes to create or delete. Set your Terraform timeouts accordingly:

timeouts {
  create = "30m"
  delete = "30m"
}

2. Subscription Association is the Key

Don't try to create subscriptions via Terraform in most cases. Instead, create them in the portal (or via billing APIs) and associate them:

resource "azurerm_management_group_subscription_association" "assignments" {
  management_group_id = azurerm_management_group.prod.id
  subscription_id     = "/subscriptions/${var.existing_subscription_id}"
}

3. Azure Policy ≠ AWS SCPs

While conceptually similar, Azure Policy is more granular but also more complex. Use the built-in policies where possible, and only create custom ones for specific compliance needs.

4. Resource Groups Are First-Class Citizens

Unlike AWS, where you might create a new account for isolation, in Azure you create a new Resource Group. This is a fundamental mindset shift.

Module Structure

Our final Terraform module structure:

infra/azure/modules/organization/
├── main.tf
├── management_groups.tf       # 4-tier CAF hierarchy
├── subscriptions.tf           # Subscription associations
├── modules/
│   ├── policy/                # 7 SOC2 policy definitions
│   │   ├── policies/
│   │   │   ├── baseline_security.json
│   │   │   ├── region_restriction.json
│   │   │   ├── require_encryption.json
│   │   │   └── ...
│   │   └── main.tf
│   ├── iam/                   # Azure AD groups + RBAC
│   │   ├── groups.tf
│   │   └── role_assignments.tf
│   ├── monitoring/            # Log Analytics, alerts
│   ├── audit/                 # Activity log export
│   ├── budget/                # Cost management
│   └── delegation/            # Defender for Cloud

Conclusion

Building a SOC2-compliant Azure infrastructure isn't about replicating AWS patterns—it's about embracing Azure's native paradigms:

Fewer subscriptions, more Resource Groups
Management Groups for policy inheritance
Azure AD as your single identity plane
Azure Policy as your compliance engine

The result? A simpler, more cost-effective, and equally secure infrastructure that works with Azure rather than against it.

Resources

Have you implemented Azure Landing Zones? I'd love to hear about your approach in the comments! 👇

This article is part of our series on building enterprise-grade cloud infrastructure. Follow for more deep dives into Terraform, cloud architecture, and compliance automation.

Tags: #azure #terraform #devops #cloud #soc2 #compliance #infrastructure

Building a Cost-Efficient Game Launcher with AWS Pre-Signed URLs

Lucas — Thu, 04 Sep 2025 08:30:17 +0000

Introduction

We’re a small game studio based in New Zealand. Like many modern studios, we built our own game launcher (think Steam or Epic Games Launcher style) that allows players to log in to our own authentication solution, using their favorite identity provider — Google, Epic, Steam, Outlook, and more. From there, they can download and update our games.

Sounds straightforward, right? Well… not quite.

The Problem

Our games and launcher binaries are stored in Amazon S3, fronted by CloudFront with a WAF for security.

The catch:

Even with CloudFront and WAF in place, the S3 buckets were still public. That meant anyone who discovered the URL could:

Download the game repeatedly.
Share direct links with others.
Drive up our CDN data transfer bill.

When you’re distributing large binaries (think tens of gigabytes per game), the cost of unrestricted downloads quickly becomes painful.

The Solution

We needed a way to control who can download and ensure downloads were tied to real authenticated users. Enter CloudFront Pre-Signed URLs.

Our infrastructure looks like this:

API Gateway – Public entry point for download requests.
Lambda Authorizer – Validates user authentication/session.
Lambda Backend – Issues short-lived CloudFront pre-signed URLs.
SQS + DLQ – Provides a dead-letter queue for failed requests or processing errors, ensuring no data is lost and enabling easier debugging and recovery.

The flow is simple:

The launcher authenticates the player.
It calls our /download-url endpoint with the requested asset ID.
The backend Lambda validates entitlement and returns a signed URL with a short TTL (60–300s).
The launcher immediately begins the download via CloudFront.
If any step fails, the event is sent to SQS DLQ for observability.

Result: only authorized, entitled users get valid download links.

Caching & TTL (what actually changes)

Caching behavior: our API response that returns the signed URL is sent with

Cache-Control: private, max-age=0.

That prevents accidental caching of the API response. The asset itself remains cacheable at the edge; the signature gates access, not caching.
TTL strategy: we keep signed URL TTLs short (60–120s for most assets). That’s long enough for the launcher to start the transfer, but short enough to reduce link leakage.

Minimal Terraform (copy-paste)

# 1) CloudFront public key (the PEM public half of your keypair)
resource "aws_cloudfront_public_key" "downloads_pk" {
  name        = "downloads-public-key"
  encoded_key = file("${path.module}/cloudfront-public-key.pem")
  comment     = "Public key for signing download URLs"
}

# 2) Key group that includes the public key
resource "aws_cloudfront_key_group" "downloads_kg" {
  name  = "downloads-key-group"
  items = [aws_cloudfront_public_key.downloads_pk.id]
}

# 3) Origin (S3 or an Origin Access Control-backed S3)
resource "aws_cloudfront_origin_access_control" "oac" {
  name                              = "downloads-oac"
  description                       = "OAC for S3 downloads"
  origin_access_control_origin_type = "s3"
  signing_behavior                  = "always"
  signing_protocol                  = "sigv4"
}

resource "aws_s3_bucket" "assets" {
  bucket = "game-assets-example"
}

# 4) Distribution with behavior that requires signed URLs
resource "aws_cloudfront_distribution" "downloads" {
  enabled             = true
  comment             = "Game downloads (signed URLs)"
  default_root_object = ""

  origins {
    domain_name              = aws_s3_bucket.assets.bucket_regional_domain_name
    origin_id                = "s3-assets"
    origin_access_control_id = aws_cloudfront_origin_access_control.oac.id
  }

  default_cache_behavior {
    target_origin_id       = "s3-assets"
    viewer_protocol_policy = "redirect-to-https"

    allowed_methods = ["GET", "HEAD"]
    cached_methods  = ["GET", "HEAD"]

    # Require signed URLs via this key group
    trusted_key_groups = [aws_cloudfront_key_group.downloads_kg.id]

    compress = true

    # Cache policy & origin request policy can be customized as needed
    cache_policy_id          = data.aws_cloudfront_cache_policy.caching_optimized.id
    origin_request_policy_id = data.aws_cloudfront_origin_request_policy.cors_s3_origin.id
  }

  restrictions {
    geo_restriction {
      restriction_type = "none"
    }
  }

  viewer_certificate {
    cloudfront_default_certificate = true
  }
}

data "aws_cloudfront_cache_policy" "caching_optimized" {
  name = "Managed-CachingOptimized"
}

data "aws_cloudfront_origin_request_policy" "cors_s3_origin" {
  name = "Managed-CORS-S3Origin"
}

Minimal Lambda signer (Python)

Here’s a small extract based on our backend that shows the core signing path: read Key-Pair-Id from env, load private key from Secrets Manager, sign a URL, return { url, expiresAt }.

import os, json, base64
from datetime import datetime, timedelta
import boto3
from cryptography.hazmat.primitives import serialization, hashes
from cryptography.hazmat.primitives.asymmetric import padding

secrets = boto3.client("secretsmanager")

def _b64url(data: bytes) -> str:
    s = base64.b64encode(data).decode("utf-8")
    return s.replace("+", "-").replace("/", "~").replace("=", "_")

def sign_url(url: str, expires_secs: int, key_pair_id: str, private_key_pem: str) -> str:
    expire_ts = int((datetime.utcnow() + timedelta(seconds=expires_secs)).timestamp())
    policy = {"Statement": [{"Resource": url, "Condition": {"DateLessThan": {"AWS:EpochTime": expire_ts}}}]}
    policy_json = json.dumps(policy, separators=(",", ":"))

    private_key = serialization.load_pem_private_key(private_key_pem.encode(), password=None)
    signature = private_key.sign(policy_json.encode(), padding.PKCS1v15(), hashes.SHA1())

    return f"{url}?Policy={_b64url(policy_json.encode())}&Signature={_b64url(signature)}&Key-Pair-Id={key_pair_id}"

def handler(event, _context):
    cf_domain = os.environ["CLOUDFRONT_DOMAIN_NAME"]         # e.g., dxxxxx.cloudfront.net
    key_pair_id = os.environ["CLOUDFRONT_PUBLIC_KEY_ID"]      # the public key ID
    secret_arn  = os.environ["CLOUDFRONT_PRIVATE_KEY_SECRET"] # PEM private key in Secrets Manager
    path        = json.loads(event.get("body","{}")).get("path", "/launcher/ReadyLauncher.exe")

    pem = secrets.get_secret_value(SecretId=secret_arn)["SecretString"]
    url = f"https://{cf_domain}{path}"
    ttl = int(os.environ.get("PRESIGNED_URL_EXPIRY", "120"))

    signed = sign_url(url, ttl, key_pair_id, pem)
    # Important: prevent caching of *API* response
    return {
        "statusCode": 200,
        "headers": {"Content-Type": "application/json",
                    "Cache-Control": "private, max-age=0",
                    "Access-Control-Allow-Origin": "*"},
        "body": json.dumps({"url": signed, "expiresAt": int(datetime.utcnow().timestamp()) + ttl})
    }

Observability: how we know it worked

CloudFront (distribution)

CacheHitRate: should trend up for game assets once hot.
BytesDownloaded: visualize by path prefix (e.g., /launcher/* vs /games/*).
Requests: also split by path to see hot objects and suspicious spikes.

API (download-url)

Count of requests to /download-url vs 2xx success rate.
Emit a correlation ID (e.g., x-request-id) from API Gateway → Lambda logs → include it in any DLQ payloads for failed attempts.

Guardrails

403 spike alarm in CloudFront: when 4xxErrorRate exceeds a baseline over N minutes.
Unsigned access attempts: create a log metric filter from CloudFront logs counting requests for asset paths missing Key-Pair-Id/Signature query params; alarm if > threshold.

AWS Cost

Did this make sense from a cost perspective? Absolutely.

Here’s what our August AWS bill looked like for this setup:

10,000 requests handled by the Pre-Signed API.
10 TB of game assets delivered through CloudFront.
Total monthly cost for the API stack: USD $47.95.

That’s less than the cost of a single AAA game… while distributing terabytes worth of player downloads over time.

Conclusion

By moving to Pre-Signed URLs, we’ve delivered a solution that is:

Secure – Only authorized users can access downloads.
Scalable – API Gateway, Lambda, and SQS scale automatically.
Cost-efficient – Infrastructure overhead is minimal compared to raw CDN transfer charges.
Manageable – Everything is deployed and maintained via Terraform IaC.

For studios and SaaS products dealing with large assets, this pattern is a proven way to reduce risk and keep bills under control — without sacrificing user experience.

✅ Next step for us: keep refining observability (tying SQS data into dashboards) and expanding entitlement logic for future titles.

Shielding Your Apps in the Cloud: Integrating CloudFront and AWS WAF with Terraform

Lucas — Wed, 24 Jan 2024 02:01:22 +0000

Introduction

In the evolving landscape of cloud computing, securing your applications is as crucial as ever. With AWS's vast array of services, it's important to leverage the right tools to protect your infrastructure effectively. In this blog post, we'll dive into how you can enhance your AWS security posture by integrating CloudFront and AWS WAF with Terraform, ensuring your applications are fortified against threats.

Understanding the Components

Terragrunt: An extension of Terraform, Terragrunt assists in managing complex infrastructure with less duplication and more efficiency. Its power lies in its ability to manage dependencies and its dry configuration approach.
CloudFront: AWS CloudFront is a content delivery network (CDN) service that speeds up the distribution of your static and dynamic web content. Beyond performance, it offers essential security features to protect your applications.
AWS WAF: The AWS Web Application Firewall (WAF) helps protect your web applications from common web exploits that could affect application availability, compromise security, or consume excessive resources.
Integrating for Security: We'll explore how these components work in unison, providing a robust defense mechanism for your applications.

Integrating Terragrunt with AWS Services

In recent blog post, we discussed how we can connect Terragrunt/Terraform with AWS.

Assuming you have that in place (or similar), I'll walk you through the steps required to deploy the services mentioned in this post as well as some additional monitoring solutions such as CloudWatch Alerts with notifications sent to Slack.

Setting Up Security Measures

Configuring AWS WAF:

A deep dive into setting up rules in AWS WAF to protect your applications from common web attacks and vulnerabilities.

Using the Cloudposse Terraform module, I have the following configuration:

inputs = {
  name  = "web-acl-cf"
  scope = "CLOUDFRONT"

  default_action = "allow"

  visibility_config = {
    cloudwatch_metrics_enabled = true
    metric_name                = "cloudfront-alb-waf"
    sampled_requests_enabled   = true
  }

  ###################
  # Logging
  ###################
  log_destination_configs = [dependency.s3.outputs.s3_bucket_arn]

  ###################
  # WAF Rules
  ###################
  managed_rule_group_statement_rules = [
    {
      name            = "AWSManagedRulesKnownBadInputsRuleSet"
      override_action = "count"
      priority        = 1

      statement = {
        name        = "AWSManagedRulesKnownBadInputsRuleSet"
        vendor_name = "AWS"
      }

      visibility_config = {
        cloudwatch_metrics_enabled = true
        sampled_requests_enabled   = true
        metric_name                = "CloudfrontAWSManagedRulesKnownBadInputsRuleSet"
      }
    },
    {
      name            = "AWSManagedRulesAmazonIpReputationList"
      override_action = "count"
      priority        = 2

      statement = {
        name        = "AWSManagedRulesAmazonIpReputationList"
        vendor_name = "AWS"
      }

      visibility_config = {
        cloudwatch_metrics_enabled = true
        sampled_requests_enabled   = true
        metric_name                = "CloudfrontAWSManagedRulesAmazonIpReputationList"
      }
    },
    {
      name            = "AWSManagedRulesBotControlRuleSet"
      override_action = "count"
      priority        = 3

      statement = {
        name        = "AWSManagedRulesBotControlRuleSet"
        vendor_name = "AWS"
      }

      visibility_config = {
        cloudwatch_metrics_enabled = true
        sampled_requests_enabled   = true
        metric_name                = "CloudfrontAWSManagedRulesBotControlRuleSet"
      }
    },
    {
      name            = "AWSManagedRulesCommonRuleSet"
      override_action = "count"
      priority        = 4

      statement = {
        name        = "AWSManagedRulesCommonRuleSet"
        vendor_name = "AWS"
      }

      visibility_config = {
        cloudwatch_metrics_enabled = true
        sampled_requests_enabled   = true
        metric_name                = "CloudfrontAWSManagedRulesCommonRuleSet"
      }
    },
    {
      name            = "AWSManagedRulesPHPRuleSet"
      override_action = "count"
      priority        = 5

      statement = {
        name        = "AWSManagedRulesPHPRuleSet"
        vendor_name = "AWS"
      }

      visibility_config = {
        cloudwatch_metrics_enabled = true
        sampled_requests_enabled   = true
        metric_name                = "CloudfrontAWSManagedRulesPHPRuleSet"
      }
    }
  ]

  rate_based_statement_rules = [
    {
      name     = "RateBasedRule"
      action   = "block"
      priority = 6

      statement = {
        limit              = local.environment_vars.locals.waf_rate_limit
        aggregate_key_type = "IP"
      }

      visibility_config = {
        cloudwatch_metrics_enabled = true
        sampled_requests_enabled   = true
        metric_name                = "CloudfrontRateBasedRule"
      }
    }
  ]

  tags = local.environment_vars.locals.tags
}

Note that the only rule that is blocking at the moment is the RateBasedRule rule.

The AWS Managed rules specified above have been taken from this official AWS documentation.

To view the AWS WAF Dashboard, make sure to select the CloudFront "region":

You'll be able to see the new rules:

You can play around with the AWS WAF rules by following their official documentation.

Leveraging CloudFront for Security:

Understanding how CloudFront can be configured to prevent DDoS attacks, bot traffic, and other types of threats.

Consuming the Terraform AWS Modules CloudFront module, I can easily deploy a CloudFront distribution in Terraform:

inputs = {
  aliases = [
    "cdn.${local.environment_vars.locals.public_domain_name}",
    "*.${local.environment_vars.locals.public_domain_name}",
  ]

  comment         = "CDN for ${local.environment_vars.locals.public_domain_name}"
  enabled         = true
  http_version    = "http1.1"
  is_ipv6_enabled = false
  web_acl_id      = dependency.waf.outputs.arn

  # When you enable additional metrics for a distribution, CloudFront sends up to 8 metrics to CloudWatch in the US East (N. Virginia) Region.
  # This rate is charged only once per month, per metric (up to 8 metrics per distribution).
  create_monitoring_subscription = local.environment_vars.locals.cloudfront_monitoring_subscription

  ####################
  # Logging
  ####################
  logging_config = {
    bucket = dependency.s3.outputs.s3_bucket_bucket_domain_name
    prefix = "${local.env}.${local.environment_vars.locals.public_domain_name}"
  }

  ####################
  # SSL certificate
  ####################
  viewer_certificate = {
    acm_certificate_arn      = dependency.acm.outputs.acm_certificate_arn
    ssl_support_method       = "sni-only"
    minimum_protocol_version = "TLSv1.2_2021"
  }

  ####################
  # Origin
  ####################
  origin = {
    alb = {
      domain_name         = local.environment_vars.locals.cloudfront_alb_alias_name
      origin_id           = "alb"
      connection_attempts = 3
      connection_timeout  = 10
      custom_origin_config = {
        http_port                = 80
        https_port               = 443
        origin_keepalive_timeout = include.root.locals.common_vars.locals.ONE_MINUTE_IN_MS
        origin_read_timeout    = include.root.locals.common_vars.locals.THREE_MINUTES_IN_MS
        origin_protocol_policy = "https-only"
        origin_ssl_protocols   = ["TLSv1.2"]
      }

      custom_header = [
        {
          name  = "X-Cloudfront-Security-Header"
          value = include.root.locals.common_vars.locals.cloudfront_security_header
        }
      ]

      origin_shield = {
        enabled              = true
        origin_shield_region = local.aws_region
      }
    }
  }

  ####################
  # Caching behavior
  ####################
  default_cache_behavior = {
    cache_policy_id = local.environment_vars.locals.cloudfront_response_headers_policy_id
    # Allow: [Headers - All viewer headers | Cookies - All | Query strings - All]
    origin_request_policy_id = "216adef6-5c7f-47e4-b989-5492eafa07d3"
    target_origin_id         = "alb"
    viewer_protocol_policy   = "redirect-to-https"
    allowed_methods          = ["DELETE", "POST", "GET", "HEAD", "OPTIONS", "PUT", "PATCH"]
    cached_methods           = ["GET", "HEAD"]
    compress                 = true
    query_string             = true
    min_ttl                  = 0
    default_ttl              = 0
    max_ttl                  = 0
    # The parameter ForwardedValues cannot be used when a cache policy is associated to the cache behavior.
    use_forwarded_values = false
  }

  ordered_cache_behavior = [
    {
      cache_policy_id = local.environment_vars.locals.cloudfront_response_headers_policy_id
      # Allow: [Headers - All viewer headers | Cookies - All | Query strings - All]
      origin_request_policy_id = "216adef6-5c7f-47e4-b989-5492eafa07d3"
      target_origin_id         = "alb"
      viewer_protocol_policy   = "redirect-to-https"
      path_pattern             = "/*"
      allowed_methods          = ["DELETE", "POST", "GET", "HEAD", "OPTIONS", "PUT", "PATCH"]
      cached_methods           = ["GET", "HEAD"]
      compress                 = true
      query_string             = true
      min_ttl                  = 0
      default_ttl              = 0
      max_ttl                  = 0
      # The parameter ForwardedValues cannot be used when a cache policy is associated to the cache behavior.
      use_forwarded_values = false
    }
  ]
}

You can see that I have the following origin: origin_id = "alb": That's my Application Load Balancer. I created a Route53 Alias (i.e. alb.example.com) and I made CF to accept that endpoint as an origin.
custom_header: In my ALB, I have rules that it will only accept traffic from CloudFront if that header is present. This is another security best practice.
In aliases, I specify that my CF distribution will respond as cdn.example.com.

Continuous Monitoring and Maintenance

AWS WAF

The WAF dashboard will give you quite a good visibility on things:

However, I do have some extra alerts going to Slack. For this integration, I use the terraform-aws-notify-slack TF module with CloudWatch.

terraform-aws-notify-slack

inputs = {
  sns_topic_name       = "notify-slack-topic"
  slack_webhook_url    = local.environment_vars.locals.slack_webhook_url
  slack_channel        = local.environment_vars.locals.notify_slack_slack_channel
  slack_username       = "reporter"
  kms_key_arn          = dependency.sns_topic_kms.outputs.key_arn
  sns_topic_kms_key_id = dependency.sns_topic_kms.outputs.key_id
  lambda_function_name = "notify-slack-${local.env}"

  lambda_description = "Lambda function which sends notifications to Slack"
  log_events         = true

  # Added this argument not to recreate the missing package once it was created
  recreate_missing_package = false

  tags = local.environment_vars.locals.tags
}

WAF Alerts

One example is:

"waf-CloudfrontRateBasedRule" = {
      alarm_name          = "waf-CloudfrontRateBasedRule-${local.env}"
      comparison_operator = "GreaterThanOrEqualToThreshold"
      evaluation_periods  = 1
      metric_name         = "BlockedRequests"
      period              = "300"
      statistic           = "Average"
      threshold           = local.environment_vars.locals.waf_rate_limit
      alarm_description   = "This metric monitors the BlockedRequests of the CloudfrontRateBasedRule WAF rule"
      alarm_actions       = [dependency.notify_slack.outputs.slack_topic_arn]
      ok_actions          = [dependency.notify_slack.outputs.slack_topic_arn]
      treat_missing_data  = "notBreaching"
      namespace           = "AWS/WAFV2"
      dimensions = {
        WebACL = dependency.waf.outputs.id
      }
    },

I have alerts in the following WAF resources:

BlockedRequests
CountedRequests

Seamless Cloud Infrastructure: Integrating Terragrunt and Terraform with AWS

Lucas — Sun, 10 Dec 2023 21:08:25 +0000

Introduction

Welcome to our latest blog post, where we delve into the world of Infrastructure as Code (IaC) using Terragrunt and Terraform, specifically focusing on their integration with Amazon Web Services (AWS). Whether you're a seasoned DevOps professional or new to cloud infrastructure, this post will guide you through the essentials of using these powerful tools in harmony with AWS.

Understanding Terragrunt and Terraform

"The cloud is someone else's computer" - Unknown

Imagine deploying a server that runs an NGINX application with only a few lines of code? This seemed impossible a couple of years ago when we were spending hours in Datacenters swapping hardware and fixing networking issues.

Terraform, an open-source infrastructure as code software tool, allows users to define and provision a cloud infrastructure using a high-level configuration language. It's a game-changer in the world of cloud computing, bringing simplicity and predictability to cloud resource management.

Terraform state

Terraform logs information about the resources it has created in a state file. This enables Terraform to know which resources are under its control and when to update and destroy them. The terraform state file, by default, is named terraform.tfstate and is held in the same directory where Terraform is run. It is created after running terraform apply.

The actual content of this file is a JSON formatted mapping of the resources defined in the configuration and those that exist in your infrastructure. When Terraform is run, it can then use this mapping to compare infrastructure to the code and make any adjustments as necessary.

Read more about the elements of Terraform architecture.

Setting Up Your Environment

If you use either Terraform or Terragrunt, you'll need to connect your IaC (Infrastructure as Code) repository to your Cloud Provider, in my case, AWS.

There are many ways of doing that, but after +8 years working with these solutions, you get to know a few tips and tricks.

My AWS account structure is as follow:

A good practice that I learnt over the years is that it can become very difficult and time consuming to manage dozens and/or hundreds of different state files in different AWS accounts. That's why I have been deploying a SharedServices account where I host all the Terraform backends and resources that it needs in order to manage infrastructure in the cloud.

These are some of the Terraform resources I store in the SharedServices account:

DynamoDB table for state locking
S3 bucket to store the state files
IAM roles and permissions
Alerts (constantly monitoring if any of the above resources are modified, deleted or even accessed)

That being said, this is how a Terraform/Terragrunt IaC repository hosted on Github will connect to AWS:

Terrateam: After getting into some really big issues when running Terragrunt with Github Actions, I decided to look for a better CI solution. Terrateam is my CI/CD tool of choice here.
- Unfortunately as of December 2023, they increased their price from USD $175 to USD $496 monthly. Me being an existing customer I still pay the old amount (thank God!)
- Alternatively, you can look at solutions like Atlantis or spacelift.
terraform-state-lock: DynamoDB table for TF state locking
devops-bucket: S3 Bucket to store TF state files
terraform-execution-role: IAM Role deployed across ALL accounts. This is the Role that TF will assume when provisioning resources in AWS

If you're provisioning the above resources for the first time, you'll have to either configure Terraform to use specific AWS keys as you won't have OIDC connection yet. In my case, I chose to have those pre-requesites resources in a CloudFormation template and deploy them with StackSets.

Deploying the pre-requesites

Github OIDC connection and role:

---
#---------------------------------------------------------------
# Deployed to the SharedServices Account
#---------------------------------------------------------------
# This template deploys the necessary resources for the Terraform Backend
AWSTemplateFormatVersion: "2010-09-09"
Description: Deploys the necessary resources for the Terraform Backend
Parameters:
  GithubRoleName:
    Type: String
    Default: "github-oidc-role"
    Description: "Name of the AWS IAM Role for Terraform assume"

Resources:
  # Creates the Github OIDC Identity Provider
  GithubOidcProvider:
    Type: AWS::IAM::OIDCProvider
    Properties:
      Url: https://token.actions.githubusercontent.com
      ClientIdList:
        - sts.amazonaws.com
        - https://github.com/github_org_name/*
      ThumbprintList:
        - 6938fd4d98bab03faadb97b34396831e3780aea1
        - 1c58a3a8518e8759bf075b76b750d4f2df264fcd

  # Creates the IAM Role that Github Actions will assume to deploy infrastrcuture
  GithubActionsRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal:
              Federated:
                - !Sub 'arn:aws:iam::${AWS::AccountId}:oidc-provider/token.actions.githubusercontent.com'
            Action:
              - 'sts:AssumeRoleWithWebIdentity'
            Condition:
              StringEquals:
                token.actions.githubusercontent.com:aud: "sts.amazonaws.com"
              StringLike:
                # https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-idp_oidc.html#idp_oidc_Create_GitHub
                token.actions.githubusercontent.com:sub: "repo:github_org_name/*"
          # https://aws.amazon.com/blogs/security/announcing-an-update-to-iam-role-trust-policy-behavior/
          - Effect: Allow
            Principal:
              AWS:
                - !Sub 'arn:aws:iam::${AWS::AccountId}:role/github-oidc-role'
            Action:
              - 'sts:AssumeRole'
      Description: Role to provide Github access to AWS
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/AdministratorAccess
      RoleName:
        Ref: GithubRoleName
      # 43200 = 12 hours
      MaxSessionDuration: 43200

NOTE: github_org_name: Change this to your Github Organization name or check this article for more details.

NOTE: Since we only want to deploy this CloudFormation template to the SharedServices account, there is no need for you to use StackSets.

Terraform backend:

---
#---------------------------------------------------------------
# Deployed to the Shared-Services account
#---------------------------------------------------------------
# This template deploys the necessary resources for the Terraform Backend
AWSTemplateFormatVersion: "2010-09-09"
Description: Deploys the necessary resources for the Terraform Backend
Parameters:
  DynamoDBTableName:
    Type: String
    Default: "terraform-state-lock"
    Description: "Name of the Terraform DynamoDB table"
  TerraformS3BucketName:
    Type: String
    Default: "devops-bucket"
    Description: "Name of the S3 Bucket for the Terraform state files"
  TerraformBackendRoleName:
    Type: String
    Default: "terraform-backend-role"
    Description: "Name of the role that users will assume for local TF development"

Resources:
  # Creates the S3 bucket that will store the Terraform State files
  TerraformS3BucketLogging:
    # checkov:skip=CKV_AWS_18: This is a logging bucket
    Type: AWS::S3::Bucket
    Properties:
      BucketName: !Join ["", [!Ref TerraformS3BucketName, "-logging"]]
      PublicAccessBlockConfiguration:
        BlockPublicAcls: true
        BlockPublicPolicy: true
        IgnorePublicAcls: true
        RestrictPublicBuckets: true
      BucketEncryption:
        ServerSideEncryptionConfiguration:
          - ServerSideEncryptionByDefault:
              SSEAlgorithm: AES256
      VersioningConfiguration:
        Status: Enabled
      Tags:
        - Key: "Backup"
          Value: "false"
        - Key: "Environment"
          Value: "SharedServices"
        - Key: "Owner"
          Value: "Terraform"

  TerraformS3BucketLoggingBucketPolicy:
    Type: AWS::S3::BucketPolicy
    Properties:
      Bucket: !Ref TerraformS3BucketLogging
      PolicyDocument:
        Version: 2012-10-17
        Statement:
          - Action:
              - "s3:List*"
              - "s3:Get*"
              - "s3:PutObject"
              - "s3:DeleteObject"
              - "s3:PutEncryptionConfiguration"
              - "s3:PutBucketPolicy"
            Effect: Allow
            Resource:
              - !Sub arn:aws:s3:::${TerraformS3BucketLogging}
              - !Sub arn:aws:s3:::${TerraformS3BucketLogging}/*
            Principal:
                AWS: !Sub '${AWS::AccountId}'

  TerraformS3Bucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: !Ref TerraformS3BucketName
      LoggingConfiguration:
        DestinationBucketName: !Ref TerraformS3BucketLogging
        LogFilePrefix: logs
      PublicAccessBlockConfiguration:
        BlockPublicAcls: true
        BlockPublicPolicy: true
        IgnorePublicAcls: true
        RestrictPublicBuckets: true
      BucketEncryption:
        ServerSideEncryptionConfiguration:
          - ServerSideEncryptionByDefault:
              SSEAlgorithm: AES256
      VersioningConfiguration:
        Status: Enabled
      Tags:
        - Key: "Backup"
          Value: "true"
        - Key: "Environment"
          Value: "SharedServices"
        - Key: "Owner"
          Value: "Terraform"

  TerraformS3BucketPolicy:
    Type: "AWS::S3::BucketPolicy"
    Properties:
      Bucket: !Ref TerraformS3Bucket
      PolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Action:
              - "s3:List*"
              - "s3:Get*"
              - "s3:PutObject"
              - "s3:DeleteObject"
              - "s3:PutEncryptionConfiguration"
              - "s3:PutBucketPolicy"
            Effect: "Allow"
            Principal:
              AWS:
                # We allow only the backend role to access the bucket
                - !Sub "arn:aws:iam::${AWS::AccountId}:role/terraform-backend-role"
            Resource:
              - !Sub arn:aws:s3:::${TerraformS3Bucket}
              - !Sub arn:aws:s3:::${TerraformS3Bucket}/*
          - Action:
              - "s3:*"
            Sid: "RootAccess"
            Effect: Allow
            Resource:
              - !Sub arn:aws:s3:::${TerraformS3Bucket}
              - !Sub arn:aws:s3:::${TerraformS3Bucket}/*
            Principal:
              AWS:
                # So that we don't lock ourselves out of the bucket
                # A specific IAM role/user should be used for this
                - !Sub "arn:aws:iam::${AWS::AccountId}:root"
          - Action:
              - "s3:*"
            Sid: "EnforcedTLS"
            Effect: Deny
            Resource:
              - !Sub arn:aws:s3:::${TerraformS3Bucket}
              - !Sub arn:aws:s3:::${TerraformS3Bucket}/*
            Principal:
              AWS: "*"
            Condition:
              Bool:
                aws:SecureTransport: "false"

  # Creates the DynamoDB Table for the TF State lock
  TerraformDynamoDBTable:
    Type: AWS::DynamoDB::Table
    Properties:
      TableName: !Ref DynamoDBTableName
      AttributeDefinitions:
        - AttributeName: "LockID"
          AttributeType: "S"
      DeletionProtectionEnabled: true
      KeySchema:
        - AttributeName: "LockID"
          KeyType: HASH
      ProvisionedThroughput:
        ReadCapacityUnits: 5
        WriteCapacityUnits: 5
      PointInTimeRecoverySpecification:
        PointInTimeRecoveryEnabled: true
      SSESpecification:
        SSEEnabled: true
        KMSMasterKeyId: alias/aws/dynamodb
        SSEType: KMS
      Tags:
        - Key: "Backup"
          Value: "true"
        - Key: "Environment"
          Value: "SharedServices"
        - Key: "Owner"
          Value: "Terraform"

  # AutoScaling rules for the DynamoDB Table
  MyTableWriteCapacityScalableTarget:
    Type: "AWS::ApplicationAutoScaling::ScalableTarget"
    DependsOn: TerraformDynamoDBTable
    Properties:
      MaxCapacity: 20
      MinCapacity: 5
      ResourceId: !Sub table/${DynamoDBTableName}
      RoleARN: !Sub arn:aws:iam::${AWS::AccountId}:role/aws-service-role/dynamodb.application-autoscaling.amazonaws.com/AWSServiceRoleForApplicationAutoScaling_DynamoDBTable
      ScalableDimension: "dynamodb:table:WriteCapacityUnits"
      ServiceNamespace: dynamodb

  MyTableWriteScalingPolicy:
    Type: "AWS::ApplicationAutoScaling::ScalingPolicy"
    Properties:
      PolicyName: WriteAutoScalingPolicy
      PolicyType: TargetTrackingScaling
      ScalingTargetId: !Ref MyTableWriteCapacityScalableTarget
      TargetTrackingScalingPolicyConfiguration:
        TargetValue: 70
        ScaleInCooldown: 60
        ScaleOutCooldown: 60
        PredefinedMetricSpecification:
          PredefinedMetricType: DynamoDBWriteCapacityUtilization

  TerraformBackendRole:
    #checkov:skip=CKV_AWS_60::Ensure IAM role allows only specific services or principals to assume it
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          # We give these roles below the ability to assume this role
          # The reason why I give the terraform-execution-role permissions to assume this role is because of local development.
          # When I run terraform locally, I want to be able to assume this role so that I can access the S3 bucket and DynamoDB table
          - Sid: "AllowAllAccounts"
            Effect: Allow
            Principal:
              AWS:
                # dev
                - arn:aws:iam::{dev_account_id}:role/terraform-execution-role
                # staging
                - arn:aws:iam::{staging_account_id}:role/terraform-execution-role
                # production
                - arn:aws:iam::{production_account_id}:role/terraform-execution-role
                # Log Archive
                - arn:aws:iam::{log-archive_account_id}:role/terraform-execution-role
                # Audit
                - arn:aws:iam::{audit_account_id}:role/terraform-execution-role
                # Master
                - arn:aws:iam::{master_account_id}:role/terraform-execution-role
                # backup
                - arn:aws:iam::{backup_account_id}:role/terraform-execution-role
                # shared-services
                - arn:aws:iam::{shared-services_account_id}:role/terraform-execution-role
                - arn:aws:iam::{shared-services_account_id}:role/github-oidc-role
            Action:
              - 'sts:AssumeRole'
          - Sid: "AllowGithubActions"
            Effect: Allow
            Principal:
              Federated:
                - !Sub 'arn:aws:iam::${AWS::AccountId}:oidc-provider/token.actions.githubusercontent.com'
            Action:
              - 'sts:AssumeRoleWithWebIdentity'
            Condition:
              StringEquals:
                token.actions.githubusercontent.com:aud: "sts.amazonaws.com"
              StringLike:
                # https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-idp_oidc.html#idp_oidc_Create_GitHub
                token.actions.githubusercontent.com:sub: "repo:github_org_name/*"
          # This is needed as we use AWS SSO and the AWSReservedSSO_DevOps* roles are used by the Engineers
          - Sid: "AllowSSOAWSAdministratorAccess"
            Effect: Allow
            Principal:
              AWS: "*"
            Action:
              - "sts:AssumeRole"
            Condition:
              ArnEquals:
                aws:PrincipalArn: "arn:aws:iam::*:role/aws-reserved/sso.amazonaws.com/*/AWSReservedSSO_DevOps*"

      Description: Role to provide Terraform access to DynamoDB and S3
      Policies:
        - PolicyName: allows-access-to-s3-policy
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Sid: "AllowS3Access"
                Effect: Allow
                Action:
                  - "s3:ListBucket"
                  - "s3:GetObject"
                  - "s3:PutObject"
                  - "s3:DeleteObject"
                Resource:
                  - !GetAtt TerraformS3Bucket.Arn
                  - !Join
                    - ""
                    - - "arn:aws:s3:::"
                      - !Ref TerraformS3Bucket
                      - /*
        - PolicyName: allows-access-to-dynamodb-policy
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Sid: "AllowDynamoDBAccess"
                Effect: Allow
                Action:
                  - "dynamodb:GetItem"
                  - "dynamodb:DeleteItem"
                  - "dynamodb:PutItem"
                  - "dynamodb:DescribeTable"
                Resource:
                  - !GetAtt TerraformDynamoDBTable.Arn
                  - !Join
                    - ""
                    - - !GetAtt TerraformDynamoDBTable.Arn
                      - /*
      RoleName: !Ref TerraformBackendRoleName

NOTE: github_org_name: Change this to your Github Organization name or check this article for more details.

NOTE: Since we only want to deploy this CloudFormation template to the SharedServices account, there is no need for you to use StackSets.

NOTE: You might notice this across some of the resources
Tags:
       - Key: "Backup"
         Value: "true"
I have AWS Backups setup which will backup anything that has a tag of Backup = true.

Connecting to AWS

Now, let's connect your Terragrunt/Terraform setup with AWS. This process involves managing AWS credentials securely and ensuring your Terraform configurations can access these credentials. This section will include detailed steps and code snippets to guide you through this process, emphasizing security and best practices.

Terrateam configuration

NOTE: More information about Terrateam's configuration here.

.terrateam/config.yml:

hooks:
  all:
    pre:
      - type: oidc
        provider: aws
        role_arn: "${TERRAFORM_GITHUB_OIDC_ROLE_ARN}"
        session_name: "terrateam"
        duration: 14400
        audience: "sts.amazonaws.com"

TERRAFORM_GITHUB_OIDC_ROLE_ARN is defined in my Github Secrets and it has the following value: arn:aws:iam::{shared-services_account_id}:role/github-oidc-role

A simple Terragrunt repository structure:

.
├── environments
│   └── test
│       ├── ap-southeast-2
│       │   ├── vpc
│       │   │   └── terragrunt.hcl
└── terragrunt.hcl

NOTE: More information about the terragrunt.hcl file can be found in this example repository.

My terragrunt.hcl file will have the backend and provider's configuration defined:

locals {
  # Automatically load region-level variables
  region_vars = read_terragrunt_config(find_in_parent_folders("region.hcl"))

  # Automatically load environment-level variables`
  environment_vars = read_terragrunt_config(find_in_parent_folders("env.hcl"))

  # Extract the variables we need for easy access
  account_name = local.environment_vars.locals.account_name
  account_id   = local.environment_vars.locals.aws_account_id
  aws_region   = local.region_vars.locals.aws_region

  # This is the S3 bucket where the Terraform State Files will be stored
  remote_state_bucket = "devops-bucket"
  # This is the DynamoDB table where Terraform will add the locking status
  dynamodb_table = "terraform-state-lock"

  # IAM Role for Terraform backend to assume
  terraform_backend_role = "arn:aws:iam::{shared-services_account_id}:role/terraform-backend-role"

  environment_path = replace(path_relative_to_include(), "environments/", "")

  # https://github.com/hashicorp/terraform/releases
  terraform_version = "latest"
  # https://github.com/gruntwork-io/terragrunt/releases
  terragrunt_version = "latest"
}

# Generate an AWS provider block
generate "provider" {
  path      = "provider.tf"
  if_exists = "overwrite_terragrunt"
  contents  = <<EOF
provider "aws" {
  region = "${local.aws_region}"

  # Only these AWS Account IDs may be operated on by this template
  allowed_account_ids = ["${local.account_id}"]

  # Assume role information
    assume_role {
        role_arn     = "arn:aws:iam::${local.account_id}:role/terraform-execution-role"
        session_name = "terragrunt-${local.account_id}-${local.aws_region}"
    }
}
EOF
}

# Configure Terragrunt to automatically store tfstate files in an S3 bucket
remote_state {
  backend = "s3"
  config = {
    encrypt        = true
    bucket         = local.remote_state_bucket
    key            = "tfstate/${path_relative_to_include()}/terraform.tfstate"
    region         = "ap-southeast-2"
    dynamodb_table = local.dynamodb_table
    role_arn       = local.terraform_backend_role
  }
  generate = {
    path      = "backend.tf"
    if_exists = "overwrite_terragrunt"
  }
}

An example of how the state files will be stored:

With that, you should be able to create the VPC resources in the Test account. :)

Conclusion

To wrap up, we'll summarize the key points covered and discuss the importance of integrating Terragrunt and Terraform with AWS for efficient cloud infrastructure management. Our goal is to equip you with the knowledge to enhance your infrastructure management practices.

Call to Action

We encourage you to share your thoughts, experiences, or questions in the comments below. And if you found this post helpful, don’t forget to follow our blog for more insightful content on cloud infrastructure and DevOps.

Smooth Sailing from AWS EC2 to ECS: A Comprehensive Migration Guide

Lucas — Mon, 04 Dec 2023 09:54:29 +0000

Introduction

Having an application running behind a Load Balancer with Auto Scaling policies is already a good choice.

With the migration from Bare metal to AWS EC2 and the upgrade of our main database from PostgreSQL 9.2 to PostgreSQL 14, we were ready to the next challenge.

Current architecture

Why Migrate to ECS?

ECS brings many benefits. Better integration with other AWS services, simplified container management and improved scalability and resource optimization are some of them.

In our case, we were using Bamboo as our CI solution to deploy application changes to our AWS environments. Each build would take around 50 minutes to complete. Imagine you're a developer and we wanted to test a change in the Test environment. You'd have to wait for ~50 minutes to see if your change succeeded or not. No, there was no local development solution in place either.

Choose your weapons!

Infrastructure as Code

There are many IaC (Infrastructure as Code) solutions out there. We decided to go with Terraform, as it's easier to manage and to find engineers who have work experience with it.

💡 Tip: Using Terraform community modules such as terraform-aws-modules and cloudposse may help you speed up the creation of those resources.

Github Actions will connect to our AWS Accounts using an OIDC connection.

Terraform state file

Managing the state of your IaC can be tricky if you have other team members creating, updating and deleting resources at the same time as you are.

Terraform has a feature that allows you to manage the state of your infrastructure using an AWS S3 bucket and DynamoDB tables. This is a huge improvement for us given the fact we had resources in CloudFormation.

CI/CD

Terraform alone is of no help. You must have a good CI solution to deploy your infrastructure changes.

We've opted to go with Github Actions, as the business was already using Github for other repositories.

Conclusion

Our cluster has been handling the traffic really well.

Improved response time just by moving platform

Before ECS migration

After ECS migration

Our new AWS architecture

PostgreSQL Live Migration from 9.2 to 14 with Bucardo

Lucas — Mon, 04 Dec 2023 08:43:29 +0000

We all hear about technical debt at some point in I.T, and often, fixing that problem is not easy. Mainly because it involves Production environments that need to be live 24/7, or simply for not having enough (human) resources.

The challenge I face is having a Production Database (1.3TB) running PostgreSQL 9.2 and Ubuntu 16.04, on EC2 instances in AWS.

version
------------------------------------------------------------------------------------------------------------------
 PostgreSQL 9.2.21 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 5.4.0-6ubuntu1~16.04.4) 5.4.0 20160609, 64-bit
(1 row)

It brings not only performance issues, but also the lack of security patches across all these years.

PostgreSQL 9.2 was released on 2012–09–10, that's almost 10 years ago. Many performance improvements have been implemented specially after PG 10, which we were not taking advantage of.

In this article, I'll talk about the challenges of migrating a live database from PostgreSQL 9.2 to 14, and improvements we've seen and where we wanna go next. :)

Database Stack

Our AWS Architecture is quite simple, deployed in two or more Availability Zones. Our database cluster had 5 instances in total, where the Master would handle all write traffic and the slaves, all the read traffic using Route53 Weighted records.

pgsql-aws-05 = Master DB running PG 9.2–100% of the write traffic
pgsql-aws-10 = Slave (Streaming Replication) DB running PG 9.2–0% of the read traffic (Mainly used for EBS Snapshots)
pgsql-aws-11 = Slave (Streaming Replication) DB running PG 9.2–25% of the read traffic
pgsql-aws-12 = Slave (Streaming Replication) DB running PG 9.2–25% of the read traffic
pgsql-aws-13 = Slave (Streaming Replication) DB running PG 9.2–25% of the read traffic
pgsql-aws-14 = Slave (Streaming Replication) DB running PG 9.2–25% of the read traffic

These instances were deployed on r4. instance types and had Ubuntu 16.04 as the OS.

The issue(s)

New Relic would constantly complain about high response times and high CPU Utilization, specially for the Master Database. We would love to be able to benefit from the Parallel query feature from PG 14, but we weren't there yet.

However, our biggest problem was with the Replication Lag. It would go up to 2 minutes at some cases, and that was causing several different issues to our customers, providing a bad user experience.

Maybe because Streaming Replication was introduced on PG 9.1, so it meant that we were using a very early version of it, without all the improvements among the years.

I did post some questions about this in the PostgreSQL General Mail list, but was unable to find a solution other than upgrading the DB version.

Setting up Bucardo

Setting up Bucardo was a challenge, as I did not have any experience with it before. Its documentation is sometimes confusing, and there isn't much information there as well.

The idea was to launch a new EC2 Instance with Bucardo in it, and make it replicate from PG 9.2 to PG 14.

Basically, the steps I did to set it up were (After following these steps):

1. Create the Bucardo Role
    `CREATE ROLE bucardo WITH LOGIN SUPERUSER PASSWORD 'password';`
2. Modify the `pg_hba.conf` file to allow connection from the Bucardo role.
    `host    replication     bucardo         10.0.0.0/16            md5`
3. Export PG roles and schema (from 9.2):
    ```


    pg_dump --schema-only --schema=public foo > /data-bucardo/foo_schemas.sql
    pg_dumpall --roles-only --database=foo > /data/foo_roles.sql


    ```
4. Import the roles and the schema into the new PG 14 DB:
    ```


    psql foo -f /data-bucardo/foo_roles.sql
    psql foo -f /data-bucardo/foo_schemas.sql


    ```
5. Create all the extensions needed in PG 14
6. Disable Triggers and FKs:
    ```


    create table if not exists foo_triggers (
        seq bigserial primary key,
        sql text
    );

    -- disable triggers
    do $$ declare t record;
    begin
        for t in select trigger_schema || '.' || event_object_table as table_name, trigger_name from information_schema.triggers
                where trigger_catalog = 'foo'
                and trigger_schema in ('public')
            loop

            insert into foo_triggers (sql) values (
                format('alter table %s disable trigger %s',
                    t.table_name, t.trigger_name));

            execute format('alter table %s disable trigger %s', t.table_name, t.trigger_name);

        end loop;
    end $$;

    create table if not exists dropped_foreign_keys_public (
            seq bigserial primary key,
            sql text
    );

    -- public schema
    do $$ declare t record;
    begin
        for t in select conrelid::regclass::varchar table_name, conname constraint_name,
                pg_catalog.pg_get_constraintdef(r.oid, true) constraint_definition
                from pg_catalog.pg_constraint r
                where r.contype = 'f'
                -- current schema only:
                and r.connamespace = (select n.oid from pg_namespace n where n.nspname = 'public')
            loop

            insert into dropped_foreign_keys_public (sql) values (
                format('alter table %s add constraint %s %s',
                    quote_ident(t.table_name), quote_ident(t.constraint_name), t.constraint_definition));

            execute format('alter table %s drop constraint %s', quote_ident(t.table_name), quote_ident(t.constraint_name));

        end loop;
    end $$;


    ```
7. Add tables and sequences to Bucardo:
    ```


    sudo bucardo add db source_db dbname=foo host=pgsql-aws-05 user=bucardouser pass='password'
    sudo bucardo add db target_db dbname=foo host=pgsql-aws-pg14 user=bucardo pass='password'

    sudo bucardo add table public.* db=source_db relgroup=foo_db_group
    sudo bucardo add sequence public.* db=source_db relgroup=foo_db_group

    sudo bucardo add dbgroup foo_db_group source_db:source target_db:target

    sudo bucardo add sync foo_sync relgroup=foo_db_group dbs=foo_db_group autokick=0

    sudo bucardo validate foo_sync


    ```
7. Export the data from PG 9.2:
    `pg_dump -Fc --data-only --disable-triggers --verbose foo > /data/foo_database_data.sql`
8. Import the data into PG 14:
    `pg_restore -d foo /data/foo_database_data.sql --disable-triggers --exit-on-error -v --jobs=15`
9. Re-enable Triggers and FKs:
    ```


    -- enable FKs
    do $$ declare t record;
            begin
                -- order by seq for easier troubleshooting when data does not satisfy FKs
                for t in select * from dropped_foreign_keys_public order by seq loop
                    execute t.sql;
                end loop;
            end $$;
    -- enable triggers
    do $$ declare t record;
    begin
        for t in select trigger_schema || '.' || event_object_table as table_name, trigger_name from information_schema.triggers
                where trigger_catalog = 'foo'
                and trigger_schema in ('public')
            loop

            execute format('alter table %s enable trigger %s', t.table_name, t.trigger_name);

        end loop;
    end $$;


    ```
10. Start Bucardo Replication:
    `sudo bucardo update sync foo_sync autokick=1`
    `sudo bucardo start`

From there, with the Bucardo replication working, we could create EBS Snapshots from the PostgreSQL 14 Master Database Instance at any time, and create slaves from those snapshots. We have been using this for the past few years and it works really good for us!

Ultimately, before moving forward with the upgrade, we want to have the same amount of Slaves as the old stack had, warm up the EBS volumes, and only after that they'll be ready to receive read-only traffic.

Failover to the new DB version

We had around 4 (four) hours of scheduled downtime to perform the failover to the new database stack.

The idea was to setup the new DB stack in a brand-new AWS LZ (Landing Zone) account managed by Control Tower. The steps we took were:

Launch an Application Load Balancer
2. Redirect all customers to the new ALB
3. From the ALB, redirect all customers to a maintenance page using `aws_lb_listener_rule`
4. Whitelist some of our IP addresses to access the application (This is for allowing few members of our team to actually access the application for testing purposes)
    ```


    resource "aws_lb_listener_rule" "https_whitelist_ip_addresses_to_app" {
    count        = terraform.workspace == "prod" ? 1 : 0
    listener_arn = aws_alb_listener.old_alb_listener_https_prod[count.index].arn

    action {
        target_group_arn = "arn:aws:elasticloadbalancing:ap-southeast-2:2____________0:targetgroup/bau-prod-elb/6_____e"
        type             = "forward"
    }

    condition {
        host_header {
            values = ["*.foo.com", "foo.com"]
        }
    }

    condition {
        source_ip {
            values = [
                "100.100.100.101/32", # Staff 1
                "100.100.100.102/32", # Staff 2
                "100.100.100.103/32", # Staff 3
                "100.100.100.104/32", # Staff 4
                ]
        }
    }}


    ```
5. Stop Bucardo
6. Modify Route53 records and point those records to the new DB cluster (Our application connects to the database using a specific endpoint)
7. Testing, testing and more testing!
8. Modify the ALB and remove the redirect rules to allow customers to access the application

The tests mentioned on Step 7 were the following:

pgdiff
[k6.io](https://www.k6.io] for stress and load tests
Manual tests using the API, Console and Mobile APPs

The new AWS Infrastructure

In the beginning of this article, I mentioned we had the database stack running on Ubuntu 16.04, which is not good. They were all deployed using CloudFormation and Bamboo, which brings us another problem. Having to manage the Bamboo on an EC2 Instance is something that we should avoid, as we would have to apply security patches, take care of backups and SSH keys rotation and more!

For that reason, when we deployed the new DB stack, we decided to go with Terraform and Bitbucket Pipelines (mainly because the company was already using Bitbucket as source of control).

The new r5. instances were all deployed using Terraform and Bitbucket Pipelines, with encrypted EBS volumes and infrastructure CIS compliant.

The gp3 EBS volumes were deployed with 10000 IOPS and 600 Mbps of throughput, as those were the numbers we thought we needed to support the traffic.

In the diagram below, you can check the current architecture, which is all managed by Terraform and Bitbucket Pipelines.

Did we solve anything at all with the upgrade?

Yes! Performance gain was amazing! Our response times dropped from ~300ms - 350ms to ~100ms - 120ms, and all this just by upgrading the DB version… no code changes and no changes to the architecture.

The new Response Time from New Relic looks (so) good - and that was taken during peak time, where we were processing more than 10,000 requests per minute.

The replication lag is gone, and it's now around 1 second or less:

System load (24 hours window) goes up to 10 during peak hours, but our r5.4xlarge master DB handles it well:

Conclusion

Overall I'm very happy with the result. Having only 4 hours of scheduled downtime for a big project like this is definitely a win, and we did not encounter any data loss or any other major problem in the following days after the upgrade.

Our EC2 cluster is performing really well, even though there are a lot to be worked on still, like slow queries improvements and changing some table designs for better performance.

I'm also very confident that we can now decrease the amount of IOPS and/or Throughput originally deployed, and potentially even move more traffic to the slaves and retire one of the slaves to save some money (at the moment each slave is getting 25% of the read traffic).

From bare metal to AWS Cloud in 90 minutes.

Lucas — Mon, 04 Dec 2023 08:26:52 +0000

Introduction

The job management platform ran a collection of bare metal servers in a leased data center. Over the last seven years, the application requirements had outgrown the capacity of the hosting platform. The company decided to migrate the application to Amazon Web Services (AWS) rather than invest in additional bare metal hardware.

The legacy environment

The job management platform ran on a collection of bare metal servers in a leased data center. Over the last seven years, the application requirements had outgrown the capacity of the hosting platform. The company decided to migrate the application to Amazon Web Services (AWS) rather than invest in additional bare metal hardware. The staff who originally set up the environment were no longer working for the company. A full-time team of three developers, as well as a sysadmin and DBA, were maintaining the legacy platform.

While the PHP code was git repository controlled, the actual sysadmin configuration was hand edited over several years; items such as Apache configuration, log file management and shared libraries. For the web and API tiers, there was a server running NGINX acting as a load balancer, distributing traffic on a round robin basis to two back-end web servers, each of which was attached to an NFS mounted disk on which the PHP code was installed.

PostgreSQL 9.2 was the database tier along with Redis for credential storage. PostgreSQL was 4TB and growing. Disk I/O was 100% on a normal day. The original designers had chosen to store media files such as images and video in the database and had created a virtual file system mapped in relational database tables within PostgreSQL. The database tier simply would not scale.

The challenges:

With over 60,000 active international users (and growing) the system was in use around the clock and the migration required minimal downtime.
Perhaps the largest problem was the old version of PostgreSQL and its +4TB of active data and growing. AWS Database Migration Service was a clear choice. It is a web service you can use to migrate data from your database that is on-premises, onto an Amazon Relational Database Service (Amazon RDS) DB instance in real time. This was not available to us, as it requires a PostgreSQL database that is version 9.3.x or later. We opted to use native PostgreSQL tools to create a live streaming replica from the legacy data center to PostgreSQL on EC2. We leveraged the flexibility of EC2 to create three tiers of cascading PostgreSQL slave databases such that when we performed the actual cut-over we promoted the primary replica to become the master and the two tiers of remaining slaves became the read replica and the fail-over databases. Using AWS snapshots and CloudFormation we can continue to build chains of cascading slaves in the event of a failure in any node, giving us an HA database solution in AWS. Due to the legacy 100% I/O issues, we provisioned the PostgreSQL EBS volumes using 10,000 IOPS and r4.4xlarge EBS–optimized instances.
The API and web tiers depended on an NFS mounted volume to share PHP code and sessions. We re-factored the code across 6 git repositories on a migration branch. The new code stored sessions in Redis and managed file uploads via S3 using an AWS SDK. Using immutable infrastructure, we were able to deploy a built AWS AMI to an auto-scaling group of m4.xlarge application servers, fronted with an AWS classic Load Balancer. We chose CloudFormation wrapped in a parameter driven script framework to build new AMIs reflecting the git commit being deployed. Part of this infrastructure as code delivered a new sysadmin environment using Ubuntu Xenial and Apache HTTP Server 2.4. This resolved the issue of the legacy sysadmin environment being unknown and hand-coded over many years.
For Orchestration, we chose Bamboo, which is a continuous integration and deployment server from Atlassian, the makers of JIRA and Confluence which were already in use by the company. We used Bamboo as a task runner/task logger to execute our parameter driven script framework which we installed on a deployment server, also built via the same CloudFormation framework.
The company had an existing skilled team with no cloud experience. AWS provided training credits and the AWS consulting partner provided skills transfer to ensure that the existing team were able to manage the migrated solution to deploy to development, UAT and production environments for each application tier.

Project results:

The main migration was completed with 90 minutes of downtime which had been communicated to the users well in advance. The migration window was over the low traffic Christmas / New Year period.

From the beginning, the job management platform ran well on AWS. For the first week or so the team was able to tune various AWS settings to achieve optimal performance.

There were problems. It was impossible to test the production workload in spite of having a full copy of the production database for extensive per-migration testing. As issues surfaced in the first hours and days, the developers and DBA were able to make and test changes quickly and deploy those changes using the Bamboo driven pipeline. We encountered no sysadmin specific issues.

Factors Critical to Project Success

The critical success factors were obvious in hindsight and key learning for any company attempting a migration project of this size. A deficit in any of the following may well have lead to an orphan project.

Senior Company Executive Backing & Strong CTO Leadership

The company CTO directed the project and provided strong leadership to ensure each participant was aligned with the same goals. He was able to communicate effectively to the board as well as dive into the detail by asking the right questions and offering the right advice at each step. He also played a vital role in communicating the company vision to AWS to achieve maximum vendor support.

Strong AWS Consulting Partner Support

The AWS Consulting Partner leads the project, using a technology stack which has undergone the test of time; there were no surprises.

Skilled and Dedicated Developers and DBA

The company employees were skilled and professional and they knew their application code well. With zero previous AWS experience, they received a one-day AWS Essentials training course and in-project training by the AWS Consulting Partner. They learned quickly and were motivated put in their best efforts. Post-migration they are successfully managing the AWS environments.

Significant Support from AWS

AWS was engaged from the beginning to help the customer directly and in partnership with the AWS Consulting Partner. The first AWS leadership principle is customer obsession. They start with the customer and work backwards. They work vigorously to earn and keep customer trust.