Forem: Sujay Pillai

I Let Three AI Agents Argue About My Architecture — Here's What Happened

Sujay Pillai — Wed, 18 Feb 2026 05:05:15 +0000

If you've ever tried to find details about a specific CNCF certification exam, you know the pain. The official CNCF training page lists dozens of courses, certifications, and workshops — all in one endless scroll. There's no way to search by exam domain, compare certification weightings side by side, or filter by difficulty level. You're left bouncing between PDFs, GitHub repos, and Linux Foundation pages just to answer a simple question like "What percentage of the CKA exam covers cluster architecture?"

I wanted to fix that — build a clean, searchable hub for all 15 CNCF certification exams. But the design space was wide open. What frontend framework? What search technology? Where to host? How to keep the data in sync with the upstream curriculum repo? The kind of project where you can burn an entire day debating "static site generator or full server?" before writing a single line of code. (Spoiler: the finished site is live at cncfexamguide.sujaypill.ai.)

Instead, I opened a terminal, typed one prompt, and watched three AI agents debate the architecture for me. Three and a half minutes later, I had a synthesized design document that was better than anything I'd have written alone — because one of the agents was specifically hired to tear the plan apart.

This is the story of that session, and a deep dive into Claude Code's Agent Teams feature — the experimental capability that lets you spawn multiple specialized agents that work in parallel, each with a distinct role and perspective.

The Challenge: A CNCF Exam Search Website

The CNCF curriculum repository contains exam materials for 15 cloud-native certifications — CKA, CKAD, CKS, KCNA, and more. Each exam has PDFs, markdown READMEs listing domains and weightings, and links to study resources.

The goal: build a clean, searchable website where someone preparing for any CNCF exam can quickly find exam domains, compare certifications, and discover resources. Sounds straightforward, but the design space is wide open. What frontend framework? What search technology? Where to host? How much infrastructure is too much?

Rather than spiraling into analysis paralysis, I decided to let Claude Code's Agent Teams feature do what it's designed for — run multiple perspectives in parallel and synthesize the results.

Assembling the Team

Agent Teams lets you launch specialized "teammates" directly from your prompt using the @teammate-name syntax. Each teammate runs as an independent Claude Code instance with its own context, tools, and instructions. Here's the prompt I used:

I'm designing a website that will run on Azure
Container Apps which will help users to search for
all the CNCF exams and its guide. The curriculum for
all the exams are found in this github repo -
https://github.com/cncf/curriculum.git
This has been cloned into current directory as
curriculum folder. Create an agent team to explore
this from different angles: one teammate on UX, one
on technical architecture, one playing devil's
advocate.

@ux-designer — Focus on information architecture,
search UX, user journeys, exam comparison features,
mobile-first, accessibility.

@architect — Focus on Azure Container Apps
deployment, frontend/backend stack, data pipeline
from GitHub, search implementation, CI/CD,
infrastructure-as-code.

@devils-advocate — Challenge every assumption.
Is ACA overkill? Are we over-engineering? Cost
analysis. Competitive landscape. What are simpler
alternatives?

The moment I hit Enter, Claude Code spawned three independent agents — each one receiving the full project context plus their specialized role instructions:

🎨 @ux-designer — Information architecture, search patterns, user journeys, exam comparison features, mobile & accessibility.
🏗️ @architect — Azure Container Apps, frontend/backend stack, data pipeline, search implementation, CI/CD, IaC with Bicep.
😈 @devils-advocate — Challenge assumptions. Is ACA overkill? Cost analysis. Simpler alternatives. What could go wrong?

Watching Them Work: The In-Process Display Mode

Here's where it gets interesting. Claude Code offers two display modes for agent teams — and the one I used, in-process mode, keeps everything in a single terminal window.

Feature	In-Process	Split Panes
Terminal setup	Single window	Requires tmux or iTerm2
Navigation	`Shift+↑/↓` to switch agents	Each agent in its own pane
Visibility	One agent at a time, status bar for all	All agents visible simultaneously
Best for	Quick sessions, any terminal	Long-running parallel work
Configuration	`--teammate-mode in-process`	`--teammate-mode split-panes`

In in-process mode, a teammate navigation bar appears at the bottom of the terminal showing all active agents. You can see each agent's name, switch between them, and monitor their progress — all without leaving your terminal.

💡 Pro Tip: Press Ctrl+T to show all teammates and their status. Press Shift+↑ to expand and read a specific agent's output. The navigation bar shows token usage and elapsed time per agent.

To configure the display mode, add this to your .claude/settings.json:

{
  "teammateMode": "in-process"
}

Or pass it as a CLI flag:

# Use in-process mode (works in any terminal)
claude --teammate-mode in-process

# Use split panes (requires tmux or iTerm2)
claude --teammate-mode split-panes

# Auto-detect: split panes if in tmux, otherwise in-process
claude --teammate-mode auto

Three Minutes, Three Perspectives

While I watched, each agent dove deep into the problem space. They independently read through the CNCF curriculum repository, analyzed the 15 exam PDFs, and produced their recommendations — all running in parallel.

0:00 – 0:30 — Agents spawn and begin reading the CNCF curriculum repository. Each one scans all 15 exam directories, PDFs, and markdown files.
0:30 – 2:00 — Parallel deep analysis. The UX designer maps user journeys. The architect designs infrastructure. The devil's advocate researches alternatives.
2:00 – 3:00 — Agents complete their analysis and report back to the main agent with detailed findings.
3:00 – 3:38 — Main agent synthesizes all three reports into a unified design document.

🏗️ The Architect's Blueprint

The @architect agent came back with a comprehensive infrastructure plan:

Key recommendations from the architect:

Data Pipeline: GitHub Actions ETL process, curriculum repo as a git submodule, Node.js scripts for markdown + PDF extraction, output as static JSON API
Search: Pagefind for client-side search — free, zero-latency, sub-100ms results
Infrastructure: Azure Container Apps with scale-to-zero, Azure Front Door CDN, Bicep IaC
Cost: ~$52-60/month at low traffic, ~$300/month at 10K concurrent users
Roadmap: MVP in 2-3 weeks, enhancements in 1-2 weeks, optimization in 1 week

😈 The Devil's Advocate Strikes

And then there was the @devils-advocate. This is where things got really interesting.

The devil's advocate didn't just nitpick — it fundamentally challenged the core assumptions:

🔥 "Is Azure Container Apps overkill for a site that's essentially serving static content? You're paying for container orchestration to serve files that change once a quarter."

"Why build a custom search solution when the data fits in a single JSON file? Pagefind works, but have you considered that a simple ctrl+F on a well-structured page might be enough for an MVP?"

"The CNCF already has a certifications page. What's your differentiation? If it's just 'better search,' that's a feature, not a product."

The Synthesis: Where the Magic Happens

Once all three agents reported back, the main agent — the orchestrator — did something I didn't expect. It didn't just concatenate the three reports. It synthesized them, identifying consensus points, resolving debates, and producing a unified design document that was stronger than any individual report.

The synthesis document had clear sections: Consensus Points (where all three agents agreed), Key Debates (where they disagreed and how the debates were resolved), and a Recommended Approach that incorporated the best ideas from each perspective.

The Plot Twist: What the Devil's Advocate Changed

This was my favorite part of the entire session. The synthesis included a dedicated section titled "What the Devil's Advocate Changed" — a list of concrete ways the critical review actually improved the final design:

Key changes driven by the devil's advocate:

Static generation over SSR — Why run a server when the data changes quarterly? The final design uses Astro as a static site generator instead of a full server-side rendered app.
Pagefind over Azure AI Search for MVP — The architect initially suggested an upgrade path to Azure AI Search. The devil's advocate argued that client-side search with Pagefind handles the dataset size (15 exams) perfectly, saving ~$200/month.
Scale-to-zero validated — The devil's advocate confirmed that Azure Container Apps' scale-to-zero was appropriate here, but pushed for a cost ceiling and monitoring alerts.
Differentiation defined — Forced the team to articulate why this site needs to exist beyond the CNCF's own page: structured comparison, cross-exam search, and study path recommendations.

This is the real power of agent teams. It's not just about parallelism — it's about productive disagreement. The devil's advocate made the architect's plan cheaper, simpler, and more focused. Without it, we'd have an over-engineered system solving the wrong problem.

Watch the Full Demo

Here's the complete 3.5-minute recording of the session described above — from the initial prompt to the final synthesis. Watch three agents analyze, debate, and converge on a design in real time:

Getting Started with Agent Teams

Agent Teams is currently an experimental feature in Claude Code. Here's how to enable it and start your first team:

Step 1: Enable the Feature

# Set the environment variable to enable agent teams
export CLAUDE_CODE_EXPERIMENTAL_AGENT_TEAMS=1

Step 2: Launch Teammates from Your Prompt

Use the @teammate-name syntax directly in your prompt. Give each teammate a clear role and specific focus areas:

Refactor the authentication module.

@security-reviewer — Audit the current auth flow for
vulnerabilities. Check token handling, session management, CSRF.

@implementer — Refactor the code to use JWT with refresh
tokens. Update all middleware and route handlers.

@tester — Write comprehensive tests for the new auth flow.
Cover edge cases: expired tokens, concurrent sessions, revocation.

Step 3: Choose Your Display Mode

# Option A: In-process (default, works everywhere)
claude --teammate-mode in-process

# Option B: Split panes (tmux or iTerm2 required)
claude --teammate-mode split-panes

# Option C: Auto-detect (recommended)
claude --teammate-mode auto

Step 4: Control Your Team

# Check status of all teammates
/teammate status

# Add a new teammate mid-session
/teammate add

# Remove a specific teammate
/teammate remove security-reviewer

# Emergency stop: press Escape to halt all agents

Best Practices

Limit to 3-5 teammates — More agents means more context and higher costs. Keep teams focused.
Give distinct roles — Overlapping responsibilities lead to redundant work. Each agent should have a clear lane.
Always include a critic — A devil's advocate or reviewer agent consistently improves outcomes by challenging assumptions.
Use for parallel independent tasks — Agent teams shine when work can be divided without constant coordination.

The Takeaway

What struck me most about this session wasn't the speed — though getting a comprehensive design in 3.5 minutes is remarkable. It was the quality of the disagreement.

In a real team, getting a UX designer, a systems architect, and a skeptical reviewer to sit in the same room and hash out a design takes days of calendar wrangling. The feedback loops are slow. The politics are real. And nobody wants to be the one who says "this is over-engineered" in front of the person who designed it.

Agent teams compress all of that into minutes. The devil's advocate has no ego to protect. The architect doesn't take the criticism personally. The UX designer's input is weighted equally. And the synthesis agent — the orchestrator — has the superhuman ability to hold all three perspectives in memory simultaneously and find the intersection.

Is this a replacement for real human collaboration? No. But as a first pass — a way to rapidly explore a design space, surface assumptions, and identify the real decisions that need human judgment — it's the most productive 3.5 minutes I've spent on architecture in a long time. And the proof is in the result — the design that came out of this session is now live at cncfexamguide.sujaypill.ai.

Resources:

docker init - create docker related assets

Sujay Pillai — Mon, 15 May 2023 16:28:43 +0000

One of the latest features that got released with Docker Desktop 4.18.0 is docker init which helps to create docker-related assets in the project folder. This post will explain how to utilize the new command in a Next.js project for generating a Dockerfile and docker-compose.yml file.

Next.js is a popular choice for building server-side rendered React applications. Its features such as automatic code splitting, server-side rendering, and static site generation have made it a powerful tool for building performant web applications.

Setup

One of the pre-requisite to work with Next.js is to have Node.js installed in your development environment. Let's see how we can scaffold the project with installing node in our local but rather making use of docker image node:18.15.0.

To create a Next.js app, open your terminal, cd into the directory you’d like to create the app in, and run the following command:

docker run -v $(pwd):/nextjsapp node:18.15.0 \
     npx create-next-app nextjsapp --use-npm \
     --example "https://github.com/vercel/next-learn/tree/master/basics/learn-starter"

npm WARN exec The following package was not found and will be installed: create-next-app@13.3.0
Creating a new Next.js app in /nextjsapp.

Downloading files from repo https://github.com/vercel/next-learn/tree/master/basics/learn-starter. This might take a moment.

Installing packages. This might take a couple of minutes.


added 20 packages, and audited 21 packages in 8s

3 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities

Success! Created nextjsapp at /nextjsapp
Inside that directory, you can run several commands:

  npm run dev
    Starts the development server.

  npm run build
    Builds the app for production.

  npm start
    Runs the built app in production mode.

We suggest that you begin by typing:

  cd nextjsapp
  npm run dev

Let us break the above command in to smaller chunks to understand what it is doing:

Since I do not have Node.js installed on my local system, I am utilizing the docker image node:18.15.0 to create a container runtime that allows me to scaffold the project within the container.
To ensure that the project location within the container is persisted, it is mounted on the host filesystem using the parameter -v $(pwd):/nextjsapp.
Once the container exits the directory from which you executed the above command will have the below files.

tree -L 1
.
├── README.md
├── node_modules
├── package-lock.json
├── package.json
├── pages
├── public
└── styles

npx create-next-app nextjsapp --use-npm \ --example "https://github.com/vercel/next- learn/tree/master/basics/learn-starter" this part of the command override CMD from the base image. npx - a cli tool to install and manage dependencies hosted in the npm registry.

Now with single docker init command you can generate Dockerfile, compose.yaml & .dockerignore file based on the framework used in your project.

docker init

Welcome to the Docker Init CLI!

This utility will walk you through creating the following files with sensible defaults for your project:
  - .dockerignore
  - Dockerfile
  - compose.yaml

Let's get started!

? What application platform does your project use? Node
? What version of Node do you want to use? 18.15.0
? Which package manager do you want to use? npm
? Do you want to run "npm run build" before starting your server? Yes
? What directory is your build output to? (comma-separate if multiple) .next
? What command do you want to use to start the app? npm start
? What port does your server listen on? 3000

CREATED: .dockerignore
CREATED: Dockerfile
CREATED: compose.yaml

✔ Your Docker files are ready!

Take a moment to review them and tailor them to your application.

When you're ready, start your application by running: docker compose up --build

Your application will be available at http://localhost:3000

Upon successful execution of the above command you will see that your Next.js project now has docker support in the project structure.

It thus helps to easily integrate Docker into my current project without the need to write Dockerfile and docker-compose file from scratch.

docker init is currently in beta. Don't use in Production environments

Ingest AWS CloudTrail Logs to Microsoft Sentinel

Sujay Pillai — Wed, 06 Apr 2022 17:46:36 +0000

Microsoft Sentinel is a Cloud Native security information and event management (SIEM) and security orchestration, automation, and response (SOAR) solution with built-in AI for analytics. It removes the cost and complexity of achieving the central and focused near real time view of the active threats in your environment.

The Data connectors page, accessible from the Microsoft Sentinel navigation menu, shows the full list of connectors that Microsoft Sentinel provides, and their status. We will use the Amazon Web Services S3 connector to pull AWS CloudTrail logs into Microsoft Sentinel.

For this connector to work we need to grant Microsoft Sentinel access to the AWS CloudTrail logs that we configured previously. By setting up this connector there is a trust relationship established between Amazon Web Services and Microsoft Sentinel. This can be achieved by creating a role that gives permission to Microsoft Sentinel to access CloudTrail logs.

In the previous blog we had already created that role with necessary permission to access CloudTrail logs.

The Role ARN and SQS Queue url in output will be handy for the connector configuration-

Changes to Outputs:
  + sentinelrole = "arn:aws:iam::123456789012:role/AzureSentinelRole"
  + sqsurl       = "https://sqs.ap-southeast-1.amazonaws.com/123456789012/awscbcloudtrailqueue"

On the Microsoft Sentinel blade navigate to Data connectors. Select Amazon Web Services S3 and in the details page click on Open connector page to configure connector.

Field	Value
ROLE ARN	arn:aws:iam::123456789012:role/AzureSentinelRole
SQS URL	https://sqs.ap-southeast-1.amazonaws.com/123456789012/awscbcloudtrailqueue

Terraform code for automating the whole setup on AWS side can be found here

You could check the status of the connector from the Connector page as shown below:

Click on AWSCloudTrail or navigate to the Log Analytics workspace to see the CloudTrail logs from your AWS Account

On successful connection Microsoft Sentinel creates a table called AWSCloudTrail with the columns as documented here

We can write custom queries using Kusto Query on top of this data and return result as shown below:

Microsoft Sentinel allows you to create custom workbooks across your data, and also comes with built-in workbook templates to allow you to quickly gain insights across your data. Once such workbook is AWS S3 Workbook built by Microsoft Sentinel Community.

SentinelHealth data table provides insights on health drifts, such as latest failure events per connector, or connectors with changes from success to failure states, which you can use to create alerts and other automated actions.

Configuring Amazon SQS queues using terraform

Sujay Pillai — Thu, 31 Mar 2022 01:54:05 +0000

Amazon SQS is a lightweight, fully-managed message queuing service. We can use SQS to decouple and scale microservices,
serverless applications, and distributed systems.
SQS makes it easy to store, receive, and send messages between software components.

In this blog you will see how we can configure an S3 bucket as source of event for a SQS Queue to be consumed by Microsoft Sentinel;a scalable, cloud-native, security information and event management (SIEM) and security orchestration, automation, and response (SOAR) solution. In our case we will showcase how we can make use of SQS to push all the CloudTrail data generated in our account to Microsoft Sentinel there by establising communication between two major cloud providers.

For this to happen we will need an IAM assumed role with necessary permissions to grant Microsoft Sentinel access to your CloudTrail logs stored in S3 Bucket and the message generated in SQS as a result of object creation in the bucket.

Resource: aws_iam_role is used to create an assumed role AzureSentinelRole to grant permissions to your Microsoft Sentinel account (ExternalId) to access your AWS resources. We also need to attach appropriate IAM permissions policies to grant Microsoft Sentinel access to the appropriate resources such as S3 bucket, SQS etc.

data "aws_iam_policy_document" "assume_role" {
  statement {
    effect  = "Allow"
    actions = ["sts:AssumeRole"]
    principals {
      type        = "AWS"
      identifiers = ["arn:aws:iam::197857026523:root"]
    }
    condition {
      test     = "StringEquals"
      variable = "sts:ExternalId"
      values   = ["65d3595c-c730-4a11-5e37-5115bae05e5e"]
    }
  }
}

resource "aws_iam_role" "this" {
  name                  = "AzureSentinelRole"
  description           = "Azure Sentinel Integration"
  assume_role_policy    = data.aws_iam_policy_document.assume_role.json
  managed_policy_arns = [
    "arn:aws:iam::aws:policy/AmazonSQSReadOnlyAccess",
    "arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess",
    "arn:aws:iam::aws:policy/service-role/AWSLambdaSQSQueueExecutionRole"
  ]
}

65d3595c-c730-4a11-5e37-5115bae05e5e : Log Analytics workspace id
197857026523 : Microsoft Sentinel's service account ID for AWS
AmazonSQSReadOnlyAccess, AWSLambdaSQSQueueExecutionRole, AmazonS3ReadOnlyAccess permission policies attached to the Sentinel role.

Resource: aws_sqs_queue is used to create the SQS queue named awscbcloudtrailqueue

Resource: aws_sqs_queue_policy is used to create SQS Policy that grants AzureSentinelRole necessary permission to carry out required actions on the newly created SQS queue.

resource "aws_sqs_queue" "sqs_queue" {
  name                      = var.trailQueueName
  delay_seconds             = 90
  max_message_size          = 2048
  message_retention_seconds = 86400
  receive_wait_time_seconds = 10
  kms_master_key_id         = aws_kms_key.primary.arn

  depends_on = [
    aws_s3_bucket.cloudtrailbucket,
    aws_kms_key.primary
  ]
}

resource "aws_sqs_queue_policy" "sqs_queue_policy" {
  queue_url = aws_sqs_queue.sqs_queue.id
  policy    = <<POLICY
{
  "Version": "2012-10-17",
  "Id": "sqspolicy",
  "Statement": [
    {
      "Sid": "CloudTrailSQS",
      "Effect": "Allow",
      "Principal": {
          "Service": "s3.amazonaws.com"
      },
      "Action": [
          "SQS:SendMessage"
      ],
      "Resource": "${aws_sqs_queue.sqs_queue.arn}",
      "Condition": {
          "ArnLike": {
              "aws:SourceArn": "${aws_s3_bucket.cloudtrailbucket.arn}"
          },
          "StringEquals": {
              "aws:SourceAccount": "${data.aws_caller_identity.current.account_id}"
          }
      }
    },
    {
      "Sid": "CloudTrailSQS",
      "Effect": "Allow",
      "Principal": {
           "AWS": "arn:aws:iam::${data.aws_caller_identity.current.account_id}:role/AzureSentinelRole"
      },
      "Action": [
        "SQS:ChangeMessageVisibility",
        "SQS:DeleteMessage",
        "SQS:ReceiveMessage",
        "SQS:GetQueueUrl"
      ],
      "Resource": "${aws_sqs_queue.sqs_queue.arn}" 
    }
  ]
}
POLICY
}

We need to configure CloudTrail S3 bucket awscbcloudtrail to send notifications to your SQS queue when an object is created in it.

Resource: aws_s3_bucket_notification is used to create a notification named awscbtrail-log-event on the bucket awscbcloudtrail with the destination as the SQS queue we created above.

resource "aws_s3_bucket_notification" "bucket_notification" {
  bucket = aws_s3_bucket.cloudtrailbucket.id
  queue {
    id        = "${var.trailName}-log-event"
    queue_arn = aws_sqs_queue.sqs_queue.arn
    events    = ["s3:ObjectCreated:*"]
  }
  depends_on = [
    aws_sqs_queue.sqs_queue
  ]
}

Once the s3 bucket notification is in place and with the proper permission set we will see the messages arriving in the queue. Shown below is the queue received 1 message -

Let's put the url for the sqs queue and the arn for the Sentinel Role that we created above as an output in terraform:

output "sentinelrole" {
    value = aws_iam_role.this.arn
}

output "sqsurl" {
  value = aws_sqs_queue.sqs_queue.url
}

....
Changes to Outputs:
  + sentinelrole = "arn:aws:iam::123456789012:role/AzureSentinelRole"
  + sqsurl       = "https://sqs.ap-southeast-1.amazonaws.com/123456789012/awscbcloudtrailqueue"

Source code for above setup is here

In the next blog we will see how we can connect Microsoft Sentinel to your AWS Account to consume the above message created in SQS queue, thus allowing us to ingest the CloudTrail data to Azure.

Encrypt CloudTrail logs with multi-region Key with Terraform

Sujay Pillai — Wed, 23 Mar 2022 02:20:03 +0000

Who did what, where and when? in my AWS account(s) through the AWS Management Console, AWS Command Line Interface, and AWS SDKs and APIs is what AWS CloudTrail answerable to you as account owner. It enables auditing, security monitoring, operational troubleshooting, records user activity and API usage across AWS services as Events. These events can be viewed from the Event History page in the AWS CloudTrail console and are available for up to 90 days after they occur.

CloudTrail records three types of events:

Management events capturing control plane actions on resources such as creating or deleting Amazon Simple Storage Service (Amazon S3) buckets.
Data events capturing data plane actions within a resource, such as reading or writing an Amazon S3 object.
Insight events shows unusual API activities in your AWS account compared to historical API usage.

By default, CloudTrail event log files are encrypted using Amazon S3 server-side encryption (SSE). You can also choose to encrypt your log files with an AWS KMS key. In the previous blog we saw how to build a multi-region key using terraform. We will make use of the same MRK to encrypt the CloudTrail log files and store it in an S3 bucket here.

Resource: aws_cloudtrail is used to create a trail for your account/organization.

resource "aws_cloudtrail" "default" {
  name                          = var.trailName
  s3_bucket_name                = var.trailBucket
  is_organization_trail         = true
  is_multi_region_trail         = true
  include_global_service_events = true
  kms_key_id                    = aws_kms_key.primary.arn
  depends_on = [
    aws_s3_bucket.cloudtrailbucket,
    aws_kms_key.primary
  ]
}

As you can see we are enabling the trail for multi-region and at organization level, which you can do it for a single-region and single account too.

For the CloudTrail to write the log-files it needs an S3 bucket and for the same reason we have depends_on added into the resource block to create the S3 bucket first.

Resource: aws_s3_bucket is used to create an S3 bucket using terraform.

resource "aws_s3_bucket" "cloudtrailbucket" {
  bucket = var.trailBucket
  depends_on = [
    aws_kms_key.primary
  ]
  force_destroy = true
  server_side_encryption_configuration {
    rule {
      apply_server_side_encryption_by_default {
        kms_master_key_id = aws_kms_key.primary.id
        sse_algorithm     = "aws:kms"
      }
      bucket_key_enabled = "false"
    }
  }
  object_lock_configuration {
    object_lock_enabled = "Enabled"
  }

  policy = <<POLICY
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AWSCloudTrailAclCheck",
            "Effect": "Allow",
            "Principal": {
              "Service": "cloudtrail.amazonaws.com"
            },
            "Action": "s3:GetBucketAcl",
            "Resource": [
                "arn:aws:s3:::${var.trailBucket}"
            ]
        },
        {
            "Sid": "AWSCloudTrailWriteAccount",
            "Effect": "Allow",
            "Principal": {
              "Service": "cloudtrail.amazonaws.com"
            },
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::${var.trailBucket}/AWSLogs/${data.aws_caller_identity.current.account_id}/*",
            "Condition": {
                "StringEquals": {
                    "s3:x-amz-acl": "bucket-owner-full-control",
                    "AWS:SourceArn" : "arn:aws:cloudtrail:ap-southeast-1:${data.aws_caller_identity.current.account_id}:trail/${var.trailName}"
                }
            }
        },
        {
            "Sid" : "AWSCloudTrailWriteOrganization",
            "Effect" : "Allow",
            "Principal" : {
                "Service" : "cloudtrail.amazonaws.com"
            },
            "Action" : "s3:PutObject",
            "Resource" : "arn:aws:s3:::${var.trailBucket}/AWSLogs/${data.aws_organizations_organization.myorg.id}/*",
            "Condition" : {
                "StringEquals" : {
                    "s3:x-amz-acl" : "bucket-owner-full-control",
                    "AWS:SourceArn" : "arn:aws:cloudtrail:ap-southeast-1:${data.aws_caller_identity.current.account_id}:trail/${var.trailName}"
                }
            }
        }
    ]
}
POLICY
}

If you just add the first statement in the bucket policy and configure your cloudtrail to write to this bucket you can see how CloudTrail logs would be beneficial in getting some valuable information. The below screenshot shows how the CloudTrail service reported InsufficientS3BucketPolicyException error while trying to create trail.

Also the CloudTrail would need explicit permission to use the KMS key to encrypt logs on behalf of specific accounts. If KMS key policy is not correctly configured for CloudTrail, CloudTrail cannot deliver logs. The IAM global condition key aws:SourceArn helps ensure that CloudTrail uses the KMS key only for this specific organizational trail what we are configuring. We would have to update the KMS policy for the key that we previously created with the below statement:

  statement {
    sid       = "Allow CloudTrail to encrypt logs"
    effect    = "Allow"
    actions   = ["kms:GenerateDataKey*"]
    resources = ["*"]
    principals {
      type        = "Service"
      identifiers = ["cloudtrail.amazonaws.com"]
    }
    condition {
      test     = "StringLike"
      variable = "kms:EncryptionContext:aws:cloudtrail:arn"
      values   = ["arn:aws:cloudtrail:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:trail/${var.trailName}"]
    }
    condition {
      test     = "StringEquals"
      variable = "aws:SourceArn"
      values   = ["arn:aws:cloudtrail:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:trail/${var.trailName}"]
    }
  }

If you run the terraform apply command with all the above configuration you will have the Trail created as shown below:

The terraform source code for above setup can be found here

Creating a multi-region key using terraform

Sujay Pillai — Thu, 17 Mar 2022 18:41:41 +0000

For organizations to encrypt their data in a cloud-native approach AWS provides a fully managed service AWS KMS, a high-performance key management system with the “pay as you go” model to lower costs and reduce their administration burden compared to self-managed hardware security module (HSM).

By choosing AWS KMS organizations get three options for encryption key management:

AWS KMS with customer or AWS-managed keys
AWS KMS with BYOK
AWS KMS with a KMS custom key store key management backed by CloudHSM

Terraform AWS provider version 3.64.0 introduced new resource aws_kms_replica_key by which we can create Customer Managed Key (CMK).

In this blog post, we will walkthrough the steps for creating a multi-region CMK using the resource aws_kms_replica_key which was introduced newly in Terraform AWS provider version 3.64.0.

Multi-Region keys come in handy for data security scenarios - Disaster recovery, Global data management, Distributed signing applications, Active-active applications that spun multiple regions.

As we need the resource type aws_kms_replica_key from Terraform AWS provider the below block helps to add this to our project. Make sure you have atleast 3.64.0 version to achieve this.

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 3.64.0"
    }
  }
}

Multi-Region keys are not global. You create a multi-Region primary key and then replicate it into regions that you select within an AWS partition. Then you manage the Multi-Region key in each region independently.

In our case we will have the primary key created in Singapore region while the replicas in Sydney and Jakarta respectively.

# Singapore
provider "aws" {
  region = "ap-southeast-1"
}

# Sydney
provider "aws" {
  alias  = "secondary"
  region = "ap-southeast-2"
}

# Jakarta
# 3.70.0 Terraform AWS Provider release will use AWS SDK v1.42.23 
# which adds ap-southeast-3 to the list of regions for the standard AWS partition.
# https://github.com/hashicorp/terraform-provider-aws/issues/22252
provider "aws" {
  alias  = "tertiary"
  region = "ap-southeast-3"
  skip_region_validation = true
}

NOTE: If you are trying to create a KMS replica in JAKARTA region you will encounter an error as below
Error: Invalid AWS Region: ap-southeast-3
with provider["registry.terraform.io/hashicorp/aws"].tertiary,

This is becuase support for ap-southeast-3 was added in AWS SDK v1.42.23 and used in Terraform AWS Provider v3.70.0. The temporary solution for this would be to add skip_region_validation = true statement in the provider block.

Unlike other AWS resource policies, a AWS KMS key policy does not automatically give permission to the account or any of its users. To give permission to account administrators, the key policy must include an explicit statement that provides this permission, like this one.

data "aws_iam_policy_document" "kms" {
  # Allow root users full management access to key
  statement {
    effect = "Allow"
    actions = [
      "kms:*"
    ]
    resources = ["*"]
    principals {
      type        = "AWS"
      identifiers = ["arn:aws:iam::${data.aws_caller_identity.current.account_id}:root"]
    }
  }

  # Allow other accounts limited access to key
  statement {
    effect = "Allow"
    actions = [
      "kms:CreateGrant",
      "kms:Encrypt",
      "kms:Decrypt",
      "kms:ReEncrypt*",
      "kms:GenerateDataKey*",
      "kms:DescribeKey",
    ]

    resources = ["*"]

    # AWS account IDs that need access to this key
    principals {
      type        = "AWS"
      identifiers = var.account_ids
    }
  }
}

Creating multi-region primary key

resource "aws_kms_key" "primary" {
  description         = "CMK for AWS CB Blog"
  enable_key_rotation = true
  policy              = data.aws_iam_policy_document.kms.json
  multi_region        = true
}

Resource :aws_kms_key is used to create a single-region OR multi-region primary KMS key.

As this is a multi-region key the id & key_id has mrk- as prefix.

terraform show -json terraform.tfstate | jq '.values.root_module.resources[0].values.id'
"mrk-01641fdcadec421f9ed2665c7d78ef9c"

terraform show -json terraform.tfstate | jq '.values.root_module.resources[0].values.key_id'
"mrk-01641fdcadec421f9ed2665c7d78ef9c"

You can also refer to KMS key using its alias. Resource : aws_kms_alias is used to create an alias.

resource "aws_kms_alias" "alias" {
  target_key_id = aws_kms_key.primary.id
  name          = format("alias/%s", lower("AWS_CB_CMK"))
}

NOTE: "name" must begin with 'alias/' and be comprised of only [a-zA-Z0-9:/_-]

Creating multi-region replica keys

resource "aws_kms_replica_key" "secondary" {
  provider = aws.secondary

  description             = "Multi-Region replica key"
  deletion_window_in_days = 7
  primary_key_arn         = aws_kms_key.primary.arn
}

resource "aws_kms_replica_key" "tertiary" {
  provider = aws.tertiary

  description             = "Multi-Region replica key"
  deletion_window_in_days = 7
  primary_key_arn         = aws_kms_key.primary.arn
}

Resource :aws_kms_replica_key is used to create a multi-region replica key. Here we are passing explicitly the provider alias (aws.secondary & aws.tertiary) to create the keys in Sydney & Jakarta region.

You need to set a waiting period of 7 (min) - 30 (max, default) days for deleting the KMS key.

We thus have a primary key in Singapore region with its replica in Sydney & Jakarta respectively.

While you implement this and in continuation of this blog we will see

How we can consume this key in encrypting an S3 bucket to configure AWS CloudTrail

Configure an SQS

Integrating Azure Sentinel to consume AWS CloudTrail data.

Source Code for above setup available here

Enable Kubernetes Metrics Server on Docker Desktop

Sujay Pillai — Mon, 20 Sep 2021 13:43:48 +0000

The steps below in this blog will help you setup Kubernetes Metrics Server on Docker Desktop which provides a standalone instance of Kubernetes running as a Docker container.

Kubernetes Metrics Server is a scalable, efficient source of container resource metrics for Kubernetes built-in autoscaling pipelines. Metrics Server collects resource metrics from Kubelets and exposes them in Kubernetes apiserver through Metrics API for use by Horizontal Pod Autoscaler and Vertical Pod Autoscaler.

Metrics Server offers:

A single deployment that works on most clusters
Scalable support up to 5,000 node clusters
Resource efficiency: Metrics Server uses 1m core of CPU and 3 MB of memory per node

You can use Metrics Server for:

CPU/Memory based horizontal autoscaling
Automatically adjusting/suggesting resources needed by containers

Prerequisites

Install Docker Desktop
Enable Kubernetes on Docker Desktop

Once you have enabled the Kubernetes on Docker Desktop, and if you run the below commands you should see messages like:

kubectl top node 
error: Metrics API not available

kubectl top pod -A
error: Metrics API not available

Metrics server isn't included with Docker Desktop's installation of Kubernetes and to install it we will have to download the latest components.yaml file from Metrics-Server releases page and open it in your text editor.

If you try to execute the command kubectl apply -f components.yaml you will see the pods get created but with some errors as highlighted below:

Add the line --kubelet-insecure-tls under the args section as shown below [L136] :

Execute the command kubectl apply -f components.yaml to apply the changes:

Now if you execute the kubectl top node & kubectl top pod -A commands you should see the output:

$ kubectl top node 
NAME             CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
docker-desktop   1310m        32%    1351Mi          71%

$ kubectl top pod -A
NAMESPACE     NAME                                     CPU(cores)   MEMORY(bytes)   
cpu-example   cpu-demo                                 1003m        1Mi             
kube-system   coredns-f9fd979d6-g2rfx                  9m           9Mi             
kube-system   coredns-f9fd979d6-wndgm                  6m           9Mi             
kube-system   etcd-docker-desktop                      35m          36Mi            
kube-system   kube-apiserver-docker-desktop            55m          325Mi           
kube-system   kube-controller-manager-docker-desktop   41m          47Mi            
kube-system   kube-proxy-s72fj                         1m           25Mi            
kube-system   kube-scheduler-docker-desktop            9m           17Mi            
kube-system   metrics-server-56c59cf9ff-jndxd          10m          14Mi            
kube-system   storage-provisioner                      4m           5Mi             
kube-system   vpnkit-controller                        1m           15Mi

You can also use Kubernetes Dashboard to view the above data (and more information) in a web UI. It allows users to manage applications running in the cluster and troubleshoot them, as well as manage the cluster itself.

To deploy Dashboard, execute following command:

$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.1.0/aio/deploy/recommended.yaml

To access Dashboard from your local workstation you must create a secure channel to your Kubernetes cluster. Run the following command:

$ kubectl proxy

Get the token for login to dashboard using the below command:

$ kubectl -n kubernetes-dashboard describe secret $(kubectl -n kubernetes-dashboard get secret |grep default-token | awk '{print $1}')

To access the HTTPS endpoint of dashboard go to:

http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/

This setting should only be used for the local Docker Desktop Kubernetes cluster, and not recommended for any hosted or production clusters.