Forem: Alejandro Velez

Here my new post using Kiro and MCP to improve and modernize my current workflow for Authorization pipeline using Identity Center and IAM Access Analyzer

Alejandro Velez — Sun, 08 Mar 2026 22:59:16 +0000

Re-imagine DevSecOps with AWS - CD applied to Authorization with IAM Identity Center and AWS IAM Access Analyzer

Alejandro Velez for AWS Community Builders ・ Mar 7

Re-imagine DevSecOps with AWS - CD applied to Authorization with IAM Identity Center and AWS IAM Access Analyzer

Alejandro Velez — Sat, 07 Mar 2026 02:37:42 +0000

Level 300

Before diving into the technical depths of permission sets and CDK pipelines, let’s take a brief journey through the story so far. Originally, our exploration began with the challenge of managing access in the cloud task that’s both critical and complex as organizations scale. In the first installment, we introduced the foundations: using AWS IAM Identity Center as the central hub for managing user identities and permissions efficiently across multiple AWS accounts. We looked at how a consistent and automated approach helps streamline authorization, reduce manual errors, and improve auditability.

The next chapter followed with practical examples illustrating how to create and deploy permission sets via Infrastructure-as-Code. Here, we learned to define reusable templates for permissions, enabling teams to replicate best practices and accelerate onboarding for new projects. Along the way, pitfalls emerged—like misconfigured policies or improper use of wildcards—which provided valuable lessons on the importance of policy validation and iterative testing in a DevSecOps environment.

Reviewing recent news, in June 2025 AWS introduced internal access findings, which spot principals in an AWS organization with access to key resources like S3 buckets, DynamoDB tables, and RDS snapshots by analyzing various policy types. Results appear on a unified dashboard for fast response or automated notifications via Amazon EventBridge, improving compliance and data security.

In February 2026, AWS added multi‑Region replication for Identity Center, allowing workforce identities and permission sets to sync across regions. This boosts resiliency, supports data residency requirements, and ensures consistent access control for global deployments.

Additionally, the integration of generative AI solutions into authorization workflows is reshaping how policies are analyzed and managed. With the introduction of AWS Kiro —an AI-powered assistant for access analysis—teams can now automatically detect policy anomalies, receive context-aware recommendations, and accelerate remediation efforts. This part will explore these innovations and demonstrate how they can be practically applied to build secure, efficient, and adaptive authorization pipelines in the modern enterprise.

The Challenge

As an AWS security specialist, you face the task of modernizing your organization's authorization management workflow. The goal is to enhance the provisioning and management of access controls across AWS environments while preserving robust multi-region replication capabilities, upholding DevSecOps best practices, and ensuring that human oversight remains integral to the process. At the same time, you must increase resiliency, proactively mitigate risks, and introduce AI-driven solutions to streamline operations and strengthen security posture.

How can AI enhance this workflow? 🧐

Integrating AI into authorization workflows—especially with tools like Kiro— dramatically enhances both efficiency and security. AI-powered assistants can automatically analyze complex IAM policies for anomalies or risky configurations, such as improper use of wildcards or misconfigured permissions, which are often the sources of failed pipeline executions and compliance gaps. By leveraging generative AI, organizations receive real-time, context-aware recommendations for policy corrections, reducing manual troubleshooting and accelerating remediation. AI solutions can also automate policy reviews, flag potential issues before deployment, and offer suggestions to align with best practices, thus minimizing human error and strengthening security posture. In addition, AI-driven analytics help teams continuously monitor and adapt policies to evolving threats, ensuring adaptive and resilient access control across global environments. Altogether, AI capabilities transform the workflow by making policy management more proactive, scalable, and responsive to both technical and regulatory demands.

Let's start by enhancing our workflow from traditional CI/CD to incorporating an** Architect in the Loop approach**.

In the Architect in the Loop (AITL) model, architects collaborate with AI systems, which generate options and analyze trade-offs. The architect reviews, contextualizes, and approves output based on technical and organizational needs. Here, AI performs most tasks but consults the architect for decisions and guidance when necessary.

The diagram depicts a multi account AWS environment with a centralized management approach. It illustrates how various services, identity providers, and automation tools interact to enforce security, governance, and continuous deployment across multiple AWS accounts.

Key Components

Accounts and Regions
• AWS Account – DevSecOps: This is the primary development and security operations account where IAM Idenity Center is delegate.
• AWS Account – Master: Acts as the central governance hub, managing permissions and policies across the organization for itself.
• AWS Account – Security Tooling Account: Hosts security focused tools and services for monitoring and analysis, managing permissions and policies across the organization for itself.
Core Services
• Key Management Service (KMS): Provides encryption and key management for sensitive data.
• CloudWatch: Monitors system and application metrics.
• Secrets Manager: Stores and rotates secrets securely.
• AWS Chatbot: Enables chat based monitoring and alerts.
• Lambda: Runs serverless functions for custom automation.
• IAM Access Analyzer: Analyzes policies for unintended access.
• S3 Reports Bucket: Stores logs and reports for auditing.
Identity and Access Management
• Identity Center and Third Party IdP icons are shown in the diagram, indicating that identity federation and centralized access management are part of the architecture.
• IAM Access Analyzer is highlighted, showing that policy analysis is performed to ensure least privilege access using validate-aws-policies python package.
CI/CD Pipeline
The CI/CD pipeline is visualized as a sequence of steps:
• CodeCommit/GitHub (source code repository)
• CodeBuild (build step, Synth,cdk pipelines self-mutate, management environment, Validate PermissionsSet))
• Manual Approval (human gate for critical deployments)
Infrastructure Deployment
• CloudFormation templates are used for both Prepare and Deploy stages, indicating that the infrastructure is provisioned through IaC.
• The Permissions Set step shows that IAM permission sets are defined and applied as part of the deployment process.
Model Context Protocol (MCP) Integration
• Kiro IDE – Agents is highlighted, indicating that an MCP compatible IDE is used to generate policies and manage custom agents.
• The AWS MCP Server provides the necessary endpoint for agents to interact with AWS services.
• The label mcproxy-for-aws@latest refers to a proxy that facilitates secure communication between agents and AWS APIs, ensuring that policy generation and SOPs (Standard Operating Procedures) are executed safely.

Workflow Summary

Source Control – Code is stored in CodeCommit/GitHub.
Build – CodeBuild compiles the code and prepares the environment.
Validation – Permissions sets are validated to ensure compliance.
Manual Approval – A human reviewer authorizes the deployment.
Deployment – CloudFormation templates are used to provision resources.
Policy Generation – The Kiro IDE and AWS MCP Server generate policies based on requirements, leveraging the mcproxy for secure API calls.

Security and Governance Highlights

• Least privilege access is enforced through IAM Access Analyzer and Permissions Set validation.
• Centralized governance is maintained via the Master account and delegated account, which oversees permissions and policies across all accounts.
• Continuous monitoring is provided by CloudWatch and Secrets Manager, ensuring operational visibility and secure secret handling.

Hands On

It is time to create!

The first step is improving the security Engineer experience like builder or operator so enable the use of smart IDE like Kiro and custom agent for context.

Here is the agent specification:

{
  "name": "secops",
  "description": "Specialized agent for managing AWS IAM Identity Center CDK project with permission sets, policies, and pipeline automation",
  "prompt": "You are an expert in AWS IAM Identity Center, AWS CDK (Python), and CI/CD pipelines. Your expertise includes:\n\n- IAM Identity Center permission sets and assignments\n- IAM Access Analyzer policy validation\n- AWS CDK infrastructure as code (Python)\n- CodePipeline and CodeBuild automation\n- Amazon Q Developer chatbot integrations (Slack/Teams)\n- Policy validation and security best practices\n- GitHub Actions and CodeCommit PR validation\n\nYou help with:\n- Creating and managing permission sets\n- Validating IAM policies with Access Analyzer\n- Troubleshooting CDK deployments\n- Configuring pipeline notifications\n- Setting up PR validation workflows\n- Implementing least privilege access\n\nAlways prioritize security, follow AWS best practices, and provide minimal, focused implementations.",
  "tools": [
    "fs_read",
    "fs_write",
    "execute_bash",
    "grep",
    "glob",
    "code",
    "use_aws",
    "aws___search_documentation",
    "aws___read_documentation",
    "web_search"
  ],
  "allowedTools": [
    "fs_read",
    "grep",
    "glob",
    "code",
    "aws___search_documentation"
  ],
  "toolsSettings": {
    "fs_write": {
      "allowedPaths": [
        "src/**",
        "project_configs/**",
        "tests/**",
        ".github/**",
        "scripts/**",
        "docs/**",
        ".kiro/**",
        "*.md",
        "*.yaml",
        "*.yml"
      ],
      "deniedPaths": [
        "cdk.out/**",
        ".venv/**",
        "node_modules/**",
        "__pycache__/**"
      ]
    },
    "execute_bash": {
      "allowedCommands": [
        "cdk synth",
        "cdk diff*",
        "cdk deploy*",
        "cdk ls",
        "pip install*",
        "python -m pytest*",
        "validate-aws-policies*",
        "git status",
        "git diff",
        "git log*",
        "aws iam*",
        "aws sso*",
        "aws identitystore*",
        "aws codepipeline*",
        "aws logs*",
        "aws s3 ls*"
      ],
      "autoAllowReadonly": true
    },
    "use_aws": {
      "allowedServices": [
        "iam",
        "sso",
        "sso-admin",
        "identitystore",
        "codepipeline",
        "codebuild",
        "s3",
        "logs",
        "cloudformation",
        "accessanalyzer",
        "secretsmanager",
        "chatbot"
      ],
      "autoAllowReadonly": true
    }
  },
  "resources": [
    "file://README.md",
    "file://project_configs/**/*.yaml",
    "file://cdk.json",
    "file://requirements.txt",
    "skill://.kiro/skills/**/SKILL.md"
  ],
  "hooks": {
    "agentSpawn": [
      {
        "command": "echo '📋 IAM Identity Center Project Context:' && echo '  - CDK Version:' $(cdk --version 2>/dev/null || echo 'Not installed') && echo '  - Python:' $(python --version 2>&1) && echo '  - AWS Profile:' ${AWS_PROFILE:-'default'} && echo '  - Project:' $(basename $(pwd))",
        "description": "Show project environment info"
      },
      {
        "command": "if [ -d 'project_configs/policies' ]; then echo '🔍 Validating IAM policies...' && validate-aws-policies -d project_configs/policies -c --quiet || echo '⚠️  Policy validation failed'; else echo 'ℹ️  No policies directory found'; fi",
        "description": "Auto-validate IAM policies with Access Analyzer"
      }
    ]
  },
  "mcpServers": {
    "aws-mcp": {
      "command": "uvx",
      "args": [
        "mcp-proxy-for-aws@latest",
        "https://aws-mcp.us-east-1.api.aws/mcp",
        "--metadata",
        "AWS_REGION=us-east-2"
      ],
      "timeout": 100000
    }
  },
  "keyboardShortcut": "ctrl+shift+i",
  "welcomeMessage": "🔐 IAM Identity Center Manager ready! I can help with:\n  • Permission sets and policy validation\n  • CDK deployments and troubleshooting\n  • Pipeline notifications (Slack/Teams)\n  • PR validation workflows\n  • Access Analyzer integration\n\nWhat would you like to work on?"
}

The key point here is the prompt for a specialized agent designed to manage AWS IAM Identity Center projects using the AWS CDK (Cloud Development Kit) with a focus on permission sets, policies, and automation for CI/CD pipelines. This agent is described as an expert in several areas:}

• AWS IAM Identity Center: Managing permission sets and assignments, which define what users and groups can access within AWS accounts.
• AWS CDK (Python): Using infrastructure-as-code techniques to automate the creation and management of AWS resources.
• CI/CD pipelines: Automating deployment workflows with CodePipeline and CodeBuild, ensuring reliable and repeatable processes.
• Policy validation: Leveraging IAM Access Analyzer to check and validate security policies, helping maintain least-privilege access and compliance.

Now the agent skills. Agent Skills

Skills are portable instruction packages that follow the open Agent Skills standard. They bundle instructions, scripts, and templates into reusable packages that Kiro can activate when relevant to your task.
Kiro supports the Agent Skills standard, so you can import skills from the community or other compatible AI tools and share your own skills across the ecosystem.

Here is an example of my skill:

---
name: iam-identity-center-project
description: "Complete guide for IAM Identity Center CDK project including permission sets, policy validation, pipeline automation, and chatbot notifications. Use when working with this project's architecture, deployment, or troubleshooting."
keywords: [iam, identity-center, permission-sets, cdk, pipeline, chatbot, policy-validation]
---

# IAM Identity Center CDK Project

## Project Overview

This CDK project manages AWS IAM Identity Center permission sets with automated policy validation, CI/CD pipelines, and chatbot notifications.

## Key Components

### 1. Permission Sets Management
- **Location**: `project_configs/environment_options/*.yaml`
- **Stack**: `src/cdkv2_manage_identity_center_stack.py`
- Define permission sets with inline policies and managed policies
- Support for principal_name (auto-resolved) or principal_id (direct)

### 2. Policy Validation
- **Tool**: `validate-aws-policies` (IAM Access Analyzer)
- **Pipeline Step**: ValidatePermissionSet in CodeBuild
- **Formats**: JSON, HTML, Markdown reports
- **Storage**: S3 bucket (labvel-secure-reports)

### 3. CI/CD Pipeline
- **Stack**: `src/cdkv2_manage_identity_center_pipeline.py`
- **Stages**: Source → Build → UpdatePipeline → ValidatePermissionSet → Deploy
- **PR Validation**: GitHub Actions + CodeCommit triggers
- **Notifications**: Amazon Q Chatbot (Slack/Teams)

### 4. Chatbot Integration
- **Location**: `src/lib/chatbot/amazon_q_chatbot.py`
- **Features**: Native CodePipeline notifications + report summaries via webhook
- **Platforms**: Slack and Microsoft Teams
- **Reports**: Lambda posts validation summaries to chat

## Source Code Structure

src/
├── cdkv2_manage_identity_center_stack.py    # Main stack - permission sets
├── cdkv2_manage_identity_center_pipeline.py # Pipeline stack
└── lib/
    ├── chatbot/
    │   ├── amazon_q_chatbot.py              # Chatbot construct
    │   └── lambda/notification_enricher.py  # Lambda for reports
    ├── sns/sns_codestart_notifications.py   # SNS notifications
    └── custom_resources/                    # Custom resources

**Note**: Use `fs_read`, `code`, or `grep` tools to explore source files on-demand.

## Quick Commands
...

With this approach the security specialist can create friendly and secure policies and check before uploading

The results

Pipeline execution:

The slack custom notification and amazon Q message

And the reports:

Conclusion

Leverage AWS's latest features—such as multi-region Identity Center replication and advanced IAM Access Analyzer tools—and integrate generative AI assistants like Kiro. This approach combines automated policy analysis, real-time anomaly detection, and context-aware remediation with architect-in-the-loop oversight, allowing AI to handle routine tasks while security professionals review and approve critical changes. The result is a resilient, adaptive, and scalable workflow that proactively reduces risk, enforces compliance, and empowers teams to respond rapidly to evolving security and regulatory demands.

For more please contact us!

✨ Alejandro Velez, Platform Engineering Latam Lead @ GFT | AWS Ambassador

Agentic AI Development with Kiro: From Zero to SaaS Platform

Alejandro Velez — Thu, 20 Nov 2025 19:30:23 +0000

Level 300

Recently, I came up with an idea inspired by my personal experience. I am a professional calisthenics athlete and frequently organize events where a recurring challenge is providing users with a seamless way to track their results. As event managers, registering and publishing results, managing competition timing, scoring systems, and different modalities are also critical concerns. On the other hand, as an AWS solutions architect and Devsecops expert, my first thought was how I could combine both worlds to create a solution that is fast, secure, and modern—without sacrificing best practices.

You're invited to explore how agentic AI development with Kiro can transform event management and SaaS platform creation. Whether you're an AWS developer, DevOps or DevSecOps engineer, SaaS architect, AI practitioner, platform engineer, technical product owner, or event manager, join us as we delve into modern cloud and AI technologies designed to streamline event management and result tracking. Discover best practices in secure, scalable, cloud-native SaaS solutions, and accelerate your development with serverless architecture and innovative AI workflows.

Prototyping with Kiro: Accelerating Development Through Serverless Best Practices

In my pursuit of optimizing both the software development lifecycle (SDLC) and platform engineering, I have been actively exploring a variety of AI workflows and practices. The goal is to enhance efficiency and innovation within the development process. However, the immediate need is to create a rapid prototype that supports my concept effectively while adhering to best practices.

To achieve this, I prioritized serverless architectures for their speed, scalability, and cost-effectiveness, ensuring that the solution remains secure and can grow alongside user demand. Among the available options, I selected Kiro as the primary tool for development. Kiro stands out by bringing structure to AI coding through spec-driven development, enabling me to produce robust prototypes swiftly while maintaining high standards in both security and scalability.

Some key features are:

Natural prompts to structured requirements
Architectural designs backed by best practices
Discrete tasks that map to requirements

Laying the Foundation for a Reliable Prototype

Creating a reliable prototype is inherently complex, especially given the possibility that the idea may gain traction and eventually move into production. In such scenarios, it is essential to minimize technical debt from the outset, ensuring that the prototype is prepared for a seamless transition to production. Therefore, the process of selecting appropriate technologies and establishing solid foundational practices becomes a critical aspect of moving from prototyping to a Minimum Viable Product (MVP).

Technical Stack and Development Practices

For the front-end, I selected a React web application to ensure a responsive and interactive user experience. The application's architecture adheres to the Twelve-Factor App methodology, which provides a framework for building scalable and maintainable software-as-a-service solutions.
To maintain high standards of reliability and security, I strictly followed AWS best practices throughout the development process. The system is designed using event-driven architecture, allowing for efficient handling of asynchronous tasks and improved scalability.

Infrastructure as Code (IaC) is implemented using the AWS Cloud Development Kit (CDK), streamlining the process of defining and managing cloud resources in a repeatable and version-controlled manner.

For continuous integration and deployment, I integrated GitHub with self-hosted runners and AWS CodeBuild. This setup enables a seamless and automated flow from code committees to deployment, supporting rapid iteration and robust DevOps workflows.

For testing purposes, I am utilizing Nova Act to facilitate the creation of UI tests with Playwright; however, further details on this topic will be covered in subsequent posts.

I use both Kiro IDE and Kiro CLI to parallelize the project task, while the IDE works in spec tasks the CLI helps me to create some backend components for infrastructure, debugging and fixing errors.

Hands On

At the beginning of the development, I created a CDK scaffold project and inserted some files with basic project rules and deep knowledge about the system:

First according to https://kiro.dev/docs/getting-started/first-project/ we create the project and setup the steering files:

Steering files give Kiro project context, so it understands your codebase, conventions, and needs. Start by selecting Generate Steering Docs in the Kiro pane. Kiro then creates steering documents in .kiro/steering/ that outline your product, tech stack, project structure, and conventions.

Some examples for steering data files:

product.md

Athleon Platform

Multi-tenant calisthenics competition management platform with role-based access control (RBAC).

## Core Functionality

- **Organizations**: Multi-tenant teams with owner/admin/member roles
- **Events**: Competition lifecycle management (draft → published → completed)
- **Athletes**: Profile management, event registration, score submission
- **Scoring**: Advanced calculation engine with multiple scoring systems
- **WODs**: Workout templates (event-scoped and global)
- **Categories**: Competition divisions (event-scoped and transversal)
- **Scheduling**: Tournament brackets and session management

## User Roles
...

tech.md

# Tech Stack

## Infrastructure

- **IaC**: AWS CDK (TypeScript)
- **Compute**: AWS Lambda (Node.js 18.x)
- **Database**: DynamoDB (single-table per domain)
- **API**: API Gateway REST API with Cognito authorizer
- **Auth**: Amazon Cognito User Pools
- **Storage**: S3 (event images)
- **Events**: EventBridge (domain event bus + central bus)
- **Deployment**: CDK deploy with esbuild bundling

## Backend

- **Runtime**: Node.js 18.x
- **SDK**: AWS SDK v3 (@aws-sdk/client-*)
- **Testing**: Jest, Vitest, fast-check (property-based testing)
- **Shared Layer**: Lambda layer at `layers/athleon-shared/nodejs`

## Frontend

- **Framework**: React 18 + Vite
- **Routing**: React Router v6
- **State**: Zustand + React Query (@tanstack/react-query)
- **Auth**: AWS Amplify v6
- **UI**: Custom components with @aws-amplify/ui-react
- **i18n**: i18next + react-i18next
- **Testing**: Vitest + React Testing Library
- **E2E**: Playwright (in e2e-tests/)

## Common Commands

### Infrastructure

...

Now, build a feature with specs:
Specs transform high-level feature ideas into detailed implementation plans through three phases:

Requirements- User stories with acceptance criteria in EARS notation
Design- Technical architecture and implementation approach
Tasks- Discrete, trackable implementation steps

For example, one feature was: Create the RBAC system to keep the roles and responsibilities for different platform users (Super Admin, athletes, organizers and spectators). This prompt generates a structure and consolidate workflow from requirements to the code:

requierements.md

# Requirements Document

## Introduction

This specification defines a role-based user onboarding system for the Athleon platform using a single Amazon Cognito User Pool. The system must support three distinct user roles (athlete, organizer, and super admin) with different access levels and onboarding workflows. Athletes and organizers can self-register through the frontend with role selection, while super admins are created exclusively through backend scripts.

## Glossary

- **Cognito User Pool**: AWS managed user directory service that handles user authentication and stores user attributes
- **Pre-signup Trigger**: AWS Lambda function invoked by Cognito before user registration is completed
- **Custom Attribute**: User-defined attribute stored in Cognito user profile (e.g., custom:role)
- **Athlete**: A user who competes in events and views leaderboards
- **Organizer**: A user who creates and manages events, categories, and WODs
- **Super Admin**: A platform administrator with full system access
- **Self-registration**: User-initiated signup process through the frontend application
- **RBAC**: Role-Based Access Control system that determines user permissions based on assigned roles
- **JWT Token**: JSON Web Token containing user claims including the custom:role attribute
- **Onboarding Flow**: The complete process from initial signup to authenticated access

design.md

## Architecture

### High-Level Components

┌─────────────────┐
│  Frontend App   │
│  (React + AWS   │
│   Amplify UI)   │
└────────┬────────┘
         │
         │ 1. Signup with role selection
         ▼
┌─────────────────────────────────────────┐
│     Amazon Cognito User Pool            │
│  ┌───────────────────────────────────┐  │
│  │  Pre-signup Lambda Trigger        │  │
│  │  - Validates role selection       │  │
│  │  - Sets custom:role attribute     │  │
│  │  - Rejects super_admin attempts   │  │
│  └───────────────────────────────────┘  │
│  ┌───────────────────────────────────┐  │
│  │  Pre-token Generation Trigger     │  │
│  │  - Adds role to JWT claims        │  │
│  └───────────────────────────────────┘  │
└─────────────────┬───────────────────────┘
                  │
                  │ 2. JWT with role claim
                  ▼
┌─────────────────────────────────────────┐
│     API Gateway + Lambda Functions      │
│  - Extract role from JWT token          │
│  - Enforce role-based permissions       │
│  - Route to appropriate services        │
└─────────────────────────────────────────┘
...

tasks.md

# Implementation Plan

- [x] 1. Update Pre-Signup Lambda Trigger
  - Modify `lambda/auth/pre-signup-trigger.js` to extract and validate role from signup request
  - Implement logic to accept 'athlete' or 'organizer' roles
  - Implement logic to reject 'super_admin' role with error
  - Default to 'athlete' for missing or invalid roles
  - Add comprehensive CloudWatch logging for all role assignments
  - _Requirements: 1.2, 1.3, 1.4, 1.5, 5.1, 5.2, 5.3, 5.4, 5.5, 8.1, 8.2_

- [x] 1.1 Write property test for pre-signup trigger
  - **Property 1: Role assignment consistency**
  - **Property 2: Super admin rejection**
  - **Property 3: Default role assignment**
  - **Validates: Requirements 1.2, 1.3, 1.4, 1.5, 2.4, 5.1, 5.2, 5.3, 5.4**
...

Here we need to extend the capabilities using Model Context Protocol (MCP) and add more context:

• Access specialized knowledge bases and documentation
• Integrate with external APIs and services
• Use domain-specific tools and utilities
• Connect to databases and cloud services

I initially integrate two MCP servers:

{
  "mcpServers": {
    "awslabs.core-mcp-server": {
      "command": "uv",
      "args": [
        "tool",
        "run",
        "--from",
        "awslabs.core-mcp-server@latest",
        "awslabs.core-mcp-server.exe"
      ],
      "env": {
        "FASTMCP_LOG_LEVEL": "ERROR",
        "AWS_PROFILE": "labvel-dev",
        "AWS_REGION": "us-east-2",
        "aws-foundation": "true",
        "solutions-architect": "true"
      },
      "disabled": false,
      "disabledTools": [
        "aws_knowledge_aws___get_regional_availability",
        "aws_knowledge_aws___list_regions",
        "pricing_get_bedrock_patterns",
        "cost_explorer_get_cost_comparison_drivers"
      ]
    },
    "playwright": {
      "command": "npx",
      "args": [
        "@playwright/mcp@latest"
      ],
      "disabled": false
    }
  }
}

The Results

I'm excited to share the outcomes from our hands-on experience building agentic AI solutions with Kiro! 😁

The following section captures the valuable lessons I've learned, the best practices I've discovered, and how thoughtful choices have helped us grow from an initial idea to a polished SaaS platform for managing calisthenics events and athlete profiles.

The IDE Project setup

Here is the review of my project setup, as you can see there are MCP, steering project files and some specs.

The Complete system architecture for tournament auto-advance

Scheduling Domain

Watch here some exception to strict DDD and boundex contex domain patterns, in this scenario I choice make read data directly between context because use rest API or service integration increase the complexity for this use case.

Scoring domain Integration

The landing page

Best Practices and lessons learned

Make sure that your scaffold project and product rules information are clear and clean.
Runs in supervised way to avoid mistakes, Kiro makes many tasks correctly but sometimes I need reject some changes because violate some rules or use bad approach to solve a technical issue.
Don’t integrate more than 50 MCP tools, this consumes context and could be slow.
Use Kiro CLI for specific troubleshooting and parallel tasks that do not modify the current tasks or files while specs are running.
Create specialized Kiro CLI agents for each SDLC domain and product domains after creating protype or MVP.
Keep flexibility and tradeoff according to your use case. Sometimes apply deep practices like DDD add more complexity however hard guardrails and guidelines are security and operational excellent.
Keep the repository clean, while prompting and refining many md files are created, some related to upgrading summaries or analysis to debug, if those files are added as context to the agent could generate confusion and reprocess.

Thank you for your time and support. Please remember to follow us for additional updates.

✨ Alejandro Velez, Platform Engineering Latam Lead @ GFT | AWS Ambassador

Using Amazon Q and custom agents for IaC

Alejandro Velez — Wed, 17 Sep 2025 03:31:17 +0000

Alejandro Velez for AWS Community Builders

Sep 17 '25

Enhancing Infrastructure as Code Development and Operations with Amazon Q, MCP, and Thoth Framework

#devops #ai #aws #iac

Comments

5 min read

Enhancing Infrastructure as Code Development and Operations with Amazon Q, MCP, and Thoth Framework

Alejandro Velez — Wed, 17 Sep 2025 03:23:41 +0000

Level 300

With each phase of digital transformation, new approaches are introduced for developing and implementing solutions. Beginning with scripting tools such as Ansible and Chef, and progressing through innovations like Terraform, CDK, Pulumi, and today’s AI-driven agentic and autonomous systems, methodologies continually evolve. Some practices become obsolete, while fresh strategies emerge—challenging engineers to adapt, innovate, and drive progress in solution creation and maintenance.

It is common for DevOps professionals to upgrade Infrastructure as Code (IaC) regularly; maintaining clean infrastructure dependencies at scale can be challenging, but many processes are now automated. Thoth framework simplifies dependency management, automates template generation and integrates seamlessly with existing workflows, reducing manual effort and minimizing errors in large-scale infrastructure projects building and managing IaC templates created with tools such as Terraform or Tofu, leveraging wrappers like Terragrunt and Terramate.

Let me show you how common tasks can be automatically accelerated using traditional approaches and modern practices with custom agents like Amazon Q.

The left side: Development

Things are constantly evolving; tasks like coding are being redefined by tools from developer assistants to agentic AI, moving us closer to fully autonomous development. Soon, writing code may seem as outdated as using an abacus, but human interaction is important and necessary for critical thinking, architecture decisions, continuous improvement and alignment with business strategies. As a cloud architect, developer, or engineer, it is essential to define the infrastructure composition with careful consideration of application-driven design and operational models. Adhering to best practice consistent with the well-architected framework and internal guidelines—is necessary to ensure optimal performance and reliability.

So, How can I do this with minimal effort, time, and resources?

Companies use internal developer platforms with blueprints and quick starts to reduce toil, lower the learning curve, and enable self-service through established paths. Developers must interact using the correct interfaces. AI agents now serve as intuitive interfaces, exposing platform capabilities via MCP and allowing for tailored agents for each SDLC task.

Let’s begin with the code. 👽

Start by creating a custom agent with Amazon Q for IaC, including platform context via MCP and a custom CLI. This approach manages tasks like infrastructure composition, compliance, scanning, and reporting, while maintaining traditional practices such as git best practices.

The following picture depicts this setup.

The system interfaces directly with AWS services via the AWS SDK and leverages OpenTofu for infrastructure provisioning, ensuring consistent and reproducible deployments across multiple environments.

You can add any complementary MCP service from list but be careful verify the source:

awslabs / mcp

Official MCP Servers for AWS

Open source MCP servers for AWS

A suite of specialized MCP servers that help you get the most out of AWS, wherever you use MCP.

Open source MCP servers for AWS

View on GitHub

modelcontextprotocol / servers

Model Context Protocol Servers

Model Context Protocol servers

This repository is a collection of reference implementations for the Model Context Protocol (MCP), as well as references to community-built servers and additional resources.

Important

If you are looking for a list of MCP servers, you can browse published servers on the MCP Registry. The repository served by this README is dedicated to housing just the small number of reference servers maintained by the MCP steering group.

Warning

The servers in this repository are intended as reference implementations to demonstrate MCP features and SDK usage. They are meant to serve as educational examples for developers building their own MCP servers, not as production-ready solutions. Developers should evaluate their own security requirements and implement appropriate safeguards based on their specific threat model and use case.

The servers in this repository showcase the versatility and extensibility of MCP, demonstrating how it can be used to give Large…

View on GitHub

Hands On

Requirements

WSL or ubuntu 24.04
python >= 3.12
thothctl >= 0.5.3
opentofu >= 1.10.6
terragrunt >= 0.88.0

Preparing the local environment

Bootstrap you environment with the necessary tools following the next steps:

a. Download and install thothctl from pypi official repository.

pipx install thothcl

b. Install amazon Q agent and Amazon Q for your IDE, terragrunt, tofu, uv, and pipx and other tools running or just runs the devtocontainers environment.

thothctl init environment  #environment for interactive mode

Select the tools according to the recommended versions. If you already have the tools installed, please run.

thothctl check environment

Creating custom Amazon Q agent (thoth agent)

Please create the AWS Builder Id

a. Use thothctl to initialize the project with the scaffold template or clone the repository.

The Custom agent configuration files are stored as JSON files in specific directories:

Project-level custom agents .amazonq/cli-agents/{agent-name}.json
Available only within the specific project directory and its subdirectories.

The Amazon Q Developer CLI searches for a custom agent by following a defined order of precedence:
• Local custom agents first - Checks for custom agents in the current working directory
• Global custom agents second - Falls back to custom agents in your home directory
• Built-in default - Uses the default agent if no custom agent is found

ℹ️ Please visit for best practices and deep knowledge: 👉 Custom Agents Management 👈

For this scenario the scaffold project template looks like:


├── LICENSE
├── README.md
├── common
│   ├── common.hcl
│   ├── common.tfvars
│   └── variables.tf
├── docs
│   └── catalog
│       ├── catalog-info.yaml
│       ├── docs
│       │   ├── general
│       │   ├── guidelines
│       │   │   ├── architecture-definition.md
│       │   │   └── iac-composition-guidelines.md
│       │   ├── images
│       │   │   ├── DiagramArchitecture.png
│       │   │   └── graph.svg
│       │   └── index.md
│       └── mkdocs.yml
├── root.hcl
└── stacks
    ├── application
    │   ├── compute
    │   │   ├── alb
    │   │   │   ├── README.md
    │   │   │   └── terragrunt.hcl
    │   │   └── asg
    │   └── storage
    │       ├── efs
    │       └── s3
    ├── foundation
    │   ├── iam
    │   │   ├── policies
    │   │   └── roles
    │   │       └── terragrunt.hcl
    │   └── network
    │       ├── security-groups
    │       └── vpc
    │           ├── README.md
    │           └── terragrunt.hcl
    ├── observability
    │   └── monitoring
    │       ├── cloudwatch
    │       └── prometheus
    └── platform
        ├── containers
        │   ├── ecr
        │   ├── eks-control-plane
        │   │   └── terragrunt.hcl
        │   └── eks-nodegroups
        └── data
            ├── elasticache
            └── rds

You can find it in:

thothforge / terragrunt_project_scaffold

Scaffold for terragrun projects using thoth framework

Terragrunt Project Scaffold

A production-ready Terragrunt template for AWS infrastructure deployment with GitOps integration and best practices.

Overview

This scaffold provides a standardized project structure for managing AWS infrastructure using Terragrunt, with built-in support for:

Multi-environment deployments
Remote state management with S3 and DynamoDB
Code quality tools (TFLint, pre-commit hooks)
GitOps workflows
Modular architecture

Project Structure

#{project_name}#/
├── .thothcf.toml              # Template configuration
├── .gitignore                 # Git ignore rules
├── .tflint.hcl               # TFLint configuration
├── .pre-commit-config.yaml   # Pre-commit hooks
├── root.hcl                  # Root Terragrunt configuration
├── common/
│   ├── common.hcl            # Common variables and provider config
│   └── variables.tf          # Shared variable definitions
├── stacks/
│   ├── foundation/           # Core infrastructure layer
│   │   ├── network/vpc/      # VPC, subnets, routing
│   │   └── iam/roles/        # Service roles and policies
│   ├── platform/             # Shared services layer
│   │   └── containers/
│   │       └── eks-control-plane/  # EKS cluster
│   ├── application/          #

…

View on GitHub

So, the custom agent provides the minimum mcp servers, context and tools. Agents can be created based on environment, technology, specialty or for specific projects for example when using a monorepo structure to store both application and infrastructure code.

Here is the baseline agent setup:

{
  "name": "thoth",
  "description": "IaC and GitOps specialist THOTH agent for IaC deployments",
  "mcpServers": {
     "thothctl": {
      "command": "thothctl",
      "args": ["mcp", "server", "--stdio"]
    },

    "git": {
       "command": "uvx",
      "args": ["mcp-server-git"],
      "timeout": 30000
    },
    "terraform": {
      "command": "docker",
      "args": [
        "run",
        "-i",
        "--rm",
        "hashicorp/terraform-mcp-server"
      ]
    },
    "awslabs.aws-diagram-mcp-server": {
      "command": "uvx",
      "args": [
        "awslabs.aws-diagram-mcp-server"
      ],
      "env": {
        "FASTMCP_LOG_LEVEL": "ERROR"
      },
      "autoApprove": [],
      "disabled": false
    }
  },
  "tools": [
    "fs_read",
    "fs_write",
    "execute_bash",
    "use_aws",
    "@git",
    "@thothctl",
    "@terraform"
  ],
  "allowedTools": [
    "fs_read",
    "use_aws",
    "@git/git_status",
    "@git/git_log",
    "@git/git_diff"
  ],
  "toolAliases": {
    "@git/git_status": "status",
    "@git/git_log": "log",
    "@git/git_diff": "diff"
  },
  "toolsSettings": {

    "fs_write": {
      "allowedPaths": [
        "stacks/**",
        "common/**",
        "modules/**",
        "*.hcl",
        "*.tf",
        "*.tfvars",
        "*.yaml",
        "*.yml",
        "*.toml",
        "*.md",
        "docs/**"
      ]
    }

  },
  "resources": [
    "file://README.md",
    "file://LICENSE",
    "file://.thothcf.toml",
    "file://root.hcl",
    "file://common/common.hcl",
    "file://common/variables.tf",
    "file://.tflint.hcl",
    "file://.pre-commit-config.yaml",
    "file://.gitignore",
    "file://stacks/**/*.hcl",
    "file://stacks/**/*.tf",
    "file://docs/**/*.md"
  ]

}

The agent has the resources definition block:

In docs/catalog/docs/guidelines we include two guidelines, one for architecture definitions and other for IaC composition guidelines.

b. Finally, start a chat with the agent in project folder and create some stacks :


➜  q chat --agent thoth
✓ terraform loaded in 1.05 s
✓ git loaded in 1.81 s
✓ awslabs.aws-diagram-mcp-server loaded in 2.02 s
✓ thothctl loaded in 2.12 s

For example:

Thanks for reading and sharing! 🤠

The next blogs offer more examples and explains how traditional and agentic AI can be combined for optimal results. 🥸

✨ Alejandro Velez, Platform Engineering Latam Lead @ GFT | AWS Ambassador

Here the 3rd part of this series!

Alejandro Velez — Mon, 04 Aug 2025 01:48:07 +0000

Alejandro Velez for AWS Community Builders

Aug 3 '25

GitOps and IaC at Scale – ArgoCD and Open Tofu – Part 3 – Hardening and Manage users

#devops #aws #terraform #gitops

Comments 1

7 min read

GitOps and IaC at Scale – ArgoCD and Open Tofu – Part 3 – Hardening and Manage users

Alejandro Velez — Sun, 03 Aug 2025 23:40:39 +0000

Level 400

Earlier blogs covered scalable delivery, architecture patterns, and code samples. This section provides additional production-ready examples and best practices using the Well Architected Framework and GitOps with ArgoCD.

Architecture Overview

Now, suppose that you must apply some custom addons for the environment, also enable on-boarding application developer’s teams and platform teams to manage their projects and applications with the least manual effort, integration with AWS ecosystem like IAM identity Center and apply best practices for ArgoCD setup. For accomplishing this the following image depicts the high-level reference architecture with the main services:

Here the new resources are ALB as an ingress gateway for Argo server, Route 53 to enable a Hosted Zone for public Domain, AWS Certificate management for SSL trough the ALB, and IAM Identity Center for SSO capabilities.

For production environments there could be a private hosted Zone and use PKI architecture internal to enable SSL access.

For other hand, *how can you be modeling the groups and access to the Argo projects and applications? *
There is not correct answer and depending on your operational model (Conway’s Law) for example, the following image depicts a simple group for each team around a workload.

Assigning permissions should follow the principle of the least privilege, separating application and project actions. For example, the application engineering team can run all actions on Applications but only watch and update the platform and addon applications.

Hands On

It’s time to create code.

First, we must modify the hub blueprint to enable argo cd best practices
❌ Don’t use Default admin for management- Disable it.
❌ Don’t use the default project
❌ Don’t use local users just if it is completely necessary for CLI or external automatons.
✅ Create Roles and assign to groups instead of individual assignments.

Now, according to Argo cd documentation the SSO is configured with IAM Identity Center.

A working Single Sign-On configuration using Identity Center (AWS SSO) has been achieved using SAML (with Dex)

To manage IAM Identity Center in AWS, applications must be created because the CreateApplicationAPI only supports custom OAuth 2.0 apps. Third-party SAML or OAuth 2.0 apps must be set up via their respective services or the AWS console. Application creation cannot be automated with Terraform, but user assignments can still be handled through IaC.

Initially, a custom Terraform module was developed to establish the ArgoCD hardening baseline. This module leverages the gitops bridge module as its foundation and incorporates SSO configuration, along with group and default role setup and baseline specifications.

The module name is terraform-helm-hardening-gitops-bridge

You can find the source code module here 👉

thothforge / terraform-helm-hardening-gitops-bridge

GitOps bridge extended module with hardennig practices and examples

Terraform Hardening GitOps Bridge Module

A comprehensive Terraform module that provides a hardened GitOps bridge for Amazon EKS clusters, implementing security best practices and enterprise-grade configurations for GitOps workflows using ArgoCD.

📖 Table of Contents

Features

Prerequisites

Architecture

Quick Start

Examples

Configuration Options

Security Best Practices

Deployment Patterns

Monitoring and Observability

Performance Optimization

Advanced Security Configuration

DNS and Ingress Configuration

Customization Options

Cost Optimization

Best Practices

Operational Procedures

FAQ

Additional Resources

Contributing

License

Support

🚀 Features

Core Capabilities

GitOps Integration: Seamless integration with ArgoCD for GitOps workflows

Security Hardening: Enterprise-grade security configurations and best practices

Multi-Repository Support: Support for addons, platform, and workloads repositories

Flexible Deployment: Single cluster or hub-spoke architecture support

Comprehensive Addons: Pre-configured essential Kubernetes addons

Security Features

RBAC Integration: Role-based access control with customizable permissions

SSO Support: Microsoft Entra ID (Azure AD) integration

Certificate Management: Automated SSL/TLS certificate provisioning

…

View on GitHub

or 👉

registry.terraform.io

The main files are: bootstrap/argocd-values.yaml where the default values are created and parser some parameters from terraform parameters like default project name, dns or argo host, kind of ingress, tags, subnets and more.

# bootstrap/argocd-values.yaml
global:
  topologySpreadConstraints:
    - maxSkew: 1
      topologyKey: "topology.kubernetes.io/hostname"
      whenUnsatisfiable: "ScheduleAnyway"
    # Default logging options used by all components
  logging:
    # -- Set the global logging format. Either: `text` or `json`
    format: text
    # -- Set the global logging level. One of: `debug`, `info`, `warn` or `error`
    level: debug



configs:
  params:
    server.insecure: true
  # SSO configuration with Entra ID
  cm:

    url: https://${argo_host}
    dex.config: |
      logger:
        level: debug
        format: json
      connectors:
      - type: saml
        id: aws
        name: "AWS IAM Identity Center"
        config:
          # You need value of Identity Center APP SAML (IAM Identity Center sign-in URL)
          ssoURL: ${sso_assertion_url} #https://portal.sso.yourregion.amazonaws.com/saml/assertion/id
          # You need `caData` _OR_ `ca`, but not both.
          #<CA cert (IAM Identity Center Certificate of Identity Center APP SAML) passed through base64 encoding>
          caData: ${ca_data_iam_app} 
          # Path to mount the secret to the dex container
          entityIssuer: https://${argo_host}/api/dex/callback
          redirectURI: https://${argo_host}/api/dex/callback
          usernameAttr: email
          emailAttr: email
          groupsAttr: groups

  rbac:
    policy.csv: |
      p, role:platform-team, applications, *, */*, allow
      p, role:platform-team, projects, *, */*, allow
      p, role:platform-team, clusters, *, *, allow
      p, role:platform-team, repositories, *, *, allow
      p, role:platform-team, certificates, *, *, allow

      p, role:platform-team, logs, get, */*, allow
      p, role:platform-team, exec, create, */*, allow
      p, role:platform-team, applicationsets, *, */*, allow
      p, role:platform-team, accounts, get, */*, allow
      p, role:platform-team, sessions, create, */*, allow
      p, role:platform-team, sessions, delete, */*, allow
      p, role:platform-team, projects, get, *, allow
      p, role:platform-team, projects, list, *, allow
      p, role:platform-team, projects, update, *, allow
      p, role:platform-team, projects, create, *, allow

      p, role:platform-team, actions, *, */*, allow
      g, ${admin_idp_group_id}, role:platform-team

    policy.default: role:deny
    policy.matchMode: glob
    scopes: '[groups, email]'
server:
  autoscaling:
    minReplicas: 2
    maxReplicas: 20
    targetCPUUtilizationPercentage: 80
    targetMemoryUtilizationPercentage: 80
  serviceAccount:
    annotations:
      eks.amazonaws.com/role-arn: "${argocd_irsa_role_arn}"
  service:
    type:  "NodePort"
  ingress:
    enabled: "${enable_argo_ingress}"
    hostname: "${argo_host}"
    controller: "aws"
    ingressClassName: "alb"
    tls: true
    aws:
      backendProtocolVersion: GRPC
      serviceType: "NodePort" #"ClusterIP"
    annotations:
      alb.ingress.kubernetes.io/group.name: argocd
      alb.ingress.kubernetes.io/backend-protocol: HTTP
      alb.ingress.kubernetes.io/healthcheck-protocol: HTTP
      alb.ingress.kubernetes.io/listen-ports: '[{"HTTP":80},{"HTTPS":443}]'
      alb.ingress.kubernetes.io/scheme: "${aws_load_balancer_type}"
      alb.ingress.kubernetes.io/security-groups: "${argo_ingress_sg}"
      alb.ingress.kubernetes.io/certificate-arn: "${acm_certificate_arn}"
      alb.ingress.kubernetes.io/ssl-policy: ELBSecurityPolicy-TLS-1-2-2017-01
      alb.ingress.kubernetes.io/ssl-redirect: "443"
      alb.ingress.kubernetes.io/healthcheck-path: /healthz
      alb.ingress.kubernetes.io/subnets: "${ingress_subnets}"
      alb.ingress.kubernetes.io/tags: "${required_tags}"
    path: /

    # -- Ingress path type. One of `Exact`, `Prefix` or `ImplementationSpecific`
    pathType: Prefix
    #extraRules:
    #- http:
    #    paths:
    #    - path: /argocd
    #      pathType: Prefix
    #      backend:
    #        service:
    #          name: 'argo-cd-argocd-server-grpc'
    #          port:
    #            name: 'https'
  ingressGrpc:
    # -- Enable an ingress resource for the Argo CD server for dedicated [gRPC-ingress]
    enabled: false #"${enable_argo_ingress}"
    hostname: "${argo_host}"
    ingressClassName: "alb"
    annotations:
      alb.ingress.kubernetes.io/group.name: argocd
      alb.ingress.kubernetes.io/backend-protocol: HTTPS
      alb.ingress.kubernetes.io/backend-protocol-version: GRPC
      alb.ingress.kubernetes.io/listen-ports: '[{"HTTP":80},{"HTTPS":443}]'
      alb.ingress.kubernetes.io/healthcheck-protocol: HTTP
      alb.ingress.kubernetes.io/target-type: "ip"
      alb.ingress.kubernetes.io/ssl-redirect: "443"
      alb.ingress.kubernetes.io/healthcheck-path: /healthz
      alb.ingress.kubernetes.io/success-codes: "0-99"
      alb.ingress.kubernetes.io/security-groups: "${argo_ingress_sg}"
      alb.ingress.kubernetes.io/certificate-arn: "${acm_certificate_arn}"
      alb.ingress.kubernetes.io/ssl-policy: ELBSecurityPolicy-TLS-1-2-2017-01
      alb.ingress.kubernetes.io/subnets: "${ingress_subnets}"
      alb.ingress.kubernetes.io/tags: "${required_tags}"
controller:
  replicas: 3
  serviceAccount:
    annotations:
      eks.amazonaws.com/role-arn: "${argocd_irsa_role_arn}"
  metrics:
    enable: true
  priorityClassName: "system-node-critical"
  podAnnotations:
    prometheus.io/scrape: true
    prometheus.io/port: 8082
    prometheus.io/path: "/metrics"
  service:
    annotations:
      prometheus.io/scrape: true
      prometheus.io/port: 8082
      prometheus.io/path: "/metrics"


repoServer:
  replicas: 2
  serviceAccount:
    annotations:
      eks.amazonaws.com/role-arn: "${argocd_irsa_role_arn}"
    metrics:
      enable: true
    priorityClassName: "system-node-critical"
    podAnnotations:
      prometheus.io/scrape: true
      prometheus.io/port: 8084
      prometheus.io/path: "/metrics"
    service:
      annotations:
        prometheus.io/scrape: true
        prometheus.io/port: 8084
        prometheus.io/path: "/metrics"


notificationsController:
  enable: true

redis-ha:
  enabled: true
  waitForVotes: false
  quorum: 2

applicationSet:
  metrics:
    enable: true
  replicas: 2
  podAnnotations:
    prometheus.io/scrape: true
    prometheus.io/port: 8085
    prometheus.io/path: "/metrics"
  service:
    annotations:
      prometheus.io/scrape: true
      prometheus.io/port: 8085
      prometheus.io/path: "/metrics"

For other hand, the argo project default template is: bootstrap/default_argproj.yaml

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: ${default_argoproj_name}
  namespace: argocd
  labels:
    argocd.argoproj.io/instance: argoprojects-platform-team
spec:
  description: Team for TI to manage Argo projects and applications
  sourceRepos: ${jsonencode(repositories)}
  destinations:
    - namespace: '*'
      server: '*'
  clusterResourceWhitelist:
    - group: '*'
      kind: '*'
  namespaceResourceWhitelist:
    - group: '*'
      kind: '*'

You can modify according to the baseline’s requirements; this is the platform project like default.

The platform addons and platform appsets just receive the parameter for default project in argoCD but keep as the beginning of this series.

Other kinds of resources for the module are security groups and default secrets, if the SSO is not enabled for platform, for admin, developers and view only users.

For other hand, the module can create a certificate and CNAME in a public hosted zone. Is usual that these resources are managed from other team, but in this setup in the same account exists a delegate hosted zone for internal use. This allows use external DNS addon to enable easy integration and register additional services in your control plane.

Sensitive data like ca data and group id could be loaded from parameter store provided for security team. Or save in secrets manager to mount the secret onto the dex container in the argocd-dex-server Deployment.

The module can be utilized with Terragrunt as follows:

#eks_control_plane-terragrunt.hcl
include "root" {
  path   = find_in_parent_folders("root.hcl")
  expose = true
}

#include "kubectl_provider" {
#  path = find_in_parent_folders("/common/additional_providers/provider_kubectl.hcl")
#}

include "k8s_helm_provider" {
  path = find_in_parent_folders("/common/additional_providers/provider_k8s_helm.hcl")
}

dependency "eks" {
  config_path = "${get_parent_terragrunt_dir("root")}/infrastructure/containers/eks_control_plane"
  mock_outputs = {
    cluster_name                       = "dummy-cluster-name"
    cluster_endpoint                   = "dummy_cluster_endpoint"
    cluster_certificate_authority_data = "dummy_cluster_certificate_authority_data"
    cluster_version                    = "1.31"
    cluster_platform_version           = "1.31"
    oidc_provider_arn                  = "dummy_arn"
    node_security_group_id = "sg-xasr18923d"
    cluster_primary_security_group_id = "sg-xasr18923d"
  }
  mock_outputs_merge_strategy_with_state = "shallow"
}
dependency "eks_role" {
  config_path = "${get_parent_terragrunt_dir("root")}/infrastructure/iam/eks_role"
  mock_outputs = {
    iam_role_arn = "arn::..."
  }
  mock_outputs_merge_strategy_with_state = "shallow"

}
dependency "vpc" {
  config_path = "${get_parent_terragrunt_dir("root")}/infrastructure/network/vpc"
  mock_outputs = {
    vpc_id = "vpc-04e3e1e302f8c8f06"
    public_subnets = [
      "subnet-0e4c5aedfc2101502",
      "subnet-0d5061f70b69eda14",
    ]
    private_subnets = [
      "subnet-0e4c5aedfc2101502",
      "subnet-0d5061f70b69eda14",
      "subnet-0d5061f70b69eda15",
    ]
  }
  mock_outputs_merge_strategy_with_state = "shallow"
}


locals {
  # Define parameters for each workspace
  env = {
    default = {

      environment = "control-plane"
      default_project = include.root.locals.environment.locals.default_project
      enable_argo_ingress = true
      sso_assertion_url = "https://portal.sso.<SSO_REGION>.amazonaws.com/saml/assertion/<id>"
      ca_data= "BASE 64 CA Data"
      argo_host_dns = {
        domain_name            = "argocd.devsecops.labvel.io"
        zone_id                = "Z101221820UZWIPQ27722"
        aws_load_balancer_type = "internet-facing"
        validation = "public"
      }
      oss_addons = {
        enable_argo_workflows = true
        #enable_foo            = true
        # you can add any addon here, make sure to update the gitops repo with the corresponding application set
      }

      addons_metadata = merge(
        include.root.locals.environment.locals.addons_repos_properties
      )


        tags = {
          Environment = "control-plane"
          Layer       = "Containers"
        }
      }

    "dev" = {

      create = true
    }
    "prod" = {

      create = true
    }
  }
  # Merge parameters
  environment_vars = contains(keys(local.env), include.root.locals.environment.locals.workspace) ? include.root.locals.environment.locals.workspace : "default"
  workspace        = merge(local.env["default"], local.env[local.environment_vars])
}


terraform {
  source ="tfr:///thothforge/hardening-gitops-bridge/helm?version=1.0.1"

}

inputs = {
  cluster_name                       = dependency.eks.outputs.cluster_name
  cluster_endpoint                   = dependency.eks.outputs.cluster_endpoint
  cluster_platform_version           = dependency.eks.outputs.cluster_platform_version
  oidc_provider_arn                  = dependency.eks.outputs.oidc_provider_arn
  node_security_group = dependency.eks.outputs.cluster_primary_security_group_id
  vpc_id         = dependency.vpc.outputs.vpc_id
  private_subnet_ids = dependency.vpc.outputs.private_subnets
  public_subnet_ids = dependency.vpc.outputs.public_subnets

  cluster_certificate_authority_data = dependency.eks.outputs.cluster_certificate_authority_data
  enable_argo_ingress = local.workspace["enable_argo_ingress"]
  argo_host_dns = local.workspace["argo_host_dns"]
  #external_dns_domain_filters = [local.workspace["argo_host_dns"]]
  sso_assertion_url = local.workspace["sso_assertion_url"]
  ca_data= local.workspace["ca_data"]

  admin_idp_group_id ="318xx789-x071-70f5-63x6-xx21233xxx33"

  tags = local.workspace["tags"]

}

Finally you can login into argocd hub control plane using the url or through AWS SSO login interface.

The upcoming post will include additional customization options and add-ons.
Thanks for reading and sharing.🙌 👽 😃

GitOps and IaC at Scale – AWS, ArgoCD, Terragrunt, and OpenTofu – Part 2 – Creating Spoke environments

Alejandro Velez — Sun, 20 Apr 2025 00:18:12 +0000

Level 400

Hi clouders, in the previous blog you can explore the considerations to deploy at scale gitops and IaC, in this post you can learn more about how to deploy spokes cluster using the GitOps bridge framework and specific use cases with AWS enterprise capabilities and security best practices.

Architecture Overview

Spoke clusters and Scenarios

According to the best practices there are many considerations and scenarios, for example:

You can have one stack or IaC repository with the definitions for each account, team or squat. It depends on your internal organizations and share responsibility model. For this scenario suppose that the model is a decentralized DevOps Operational Model, and each squat has their custom IaC for each project. Consider keeping clear separation and isolation between environments and make match your platform strategy, for example, some organizations have a single hub for managing dev, QA and prod cluster, others have a hub to manage each environment cluster and other, and others have one hub for previous environments and another for production.
Another key point to note is the capacity planning and applications by team, some organizations allow share the same cluster for an organizational unit and keep the applications grouped by namespaces in each environment, others prefer to have one environment and cluster by workload or application. Considering the networking and security considerations has main pain and challenges for both scenarios. In this series the main assumption is that each workload has a dedicated cluster and account by environment but there is a transversal platform team that manages the cluster configuration and control plane. The following table describe the relationship between scenario and AWS accounts:

Scenario	AWS Account	Clusters
Single Hub – N Spokes	1 for HUB – N account by environment	1 cluster Hub, N Spoke Clusters by environment
M Hub - N spokes	M accounts By Hub environments – N account by environments	M Hub Clusters, N Environment Clusters

The Figure 1 depicts the architecture for this scenario.

Figure 1. GitOps bridge Deployment architecture in AWS

The FinOps practices are key point independent of your resources distribution you must consider what is the best strategy for track cost and shared resources.

Hands On

First, modify the hub infrastructure stacks to add the stack to manage the credentials cross account and allow the CI infrastructure agents to take them to register the cluster in the control plane. Also, create the role for Argocd to enable the authentication between Argocd hub and spoke cluster.

Updating Control Plane Infrastructure

So, let’s set up the credentials according to the best practices and the scenario described in the previous section, the cluster credentials are stored in parameter store and share with the organizational units for each team or business unit.

You must enable RAM as trusted service in your organization from Organizations management account and RAM Console.

For this task, a local module terraform-aws-parameter-store was created:

└── terraform-aws-ssm-parameter-sotre
    ├── README.md
    ├── data.tf
    ├── main.tf
    ├── outputs.tf
    └── variables.tf

The module creates the parameter and shares with organization id, organizational unit, or account id principals.

Now, using terragrunt the module is called to create a new stack or terragrunt unit.

#parameter_store-terragrunt.hcl

include "root" {
  path = find_in_parent_folders("root.hcl")
  expose = true
}

dependency "eks" {
  config_path = "${get_parent_terragrunt_dir("root")}/infrastructure/containers/eks_control_plane"
  mock_outputs = {
    cluster_name = "dummy-cluster-name"
    cluster_endpoint = "dummy_cluster_endpoint"
    cluster_certificate_authority_data = "dummy_cluster_certificate_authority_data"
    cluster_version = "1.31"
    cluster_platform_version = "1.31"
    oidc_provider_arn =  "dummy_arn"
    cluster_arn = "arn:aws:eks:us-east-2:105171185823:cluster/gitops-scale-dev-hub"
  }
  mock_outputs_merge_strategy_with_state = "shallow"
}

locals {
  # Define parameters for each workspace
  env = {
    default = {

      parameter_name = "/control_plane/${include.root.locals.environment.locals.workspace}/credentials"
      sharing_principals = ["ou-w3ow-k24p2opx"]
      tags = {
        Environment = "control-plane"
        Layer       = "Operations"
      }
    }
    "dev" = {

      create = true
    }
    "prod" = {

      create = true
    }
  }
  # Merge parameters
  environment_vars = contains(keys(local.env), include.root.locals.environment.locals.workspace) ? include.root.locals.environment.locals.workspace : "default"
  workspace = merge(local.env["default"], local.env[local.environment_vars])
}


terraform {
  source = "../../../modules/terraform-aws-ssm-parameter-sotre"

}

inputs = {
  parameter_name = "${local.workspace["parameter_name"]}"
  parameter_description = "Control plane credentials"
  parameter_type = "SecureString"
  parameter_tier = "Advanced"
  create_kms = true
  enable_sharing = true
  sharing_principals= local.workspace["sharing_principals"]
  parameter_value = jsonencode({
    cluster_name                       = dependency.eks.outputs.cluster_name,
    cluster_endpoint                   = dependency.eks.outputs.cluster_endpoint,
    cluster_certificate_authority_data = dependency.eks.outputs.cluster_certificate_authority_data,
    cluster_version                    = dependency.eks.outputs.cluster_version,
    cluster_platform_version           = dependency.eks.outputs.cluster_platform_version,
    oidc_provider_arn                  = dependency.eks.outputs.oidc_provider_arn,
    hub_account_id = split(":", dependency.eks.outputs.cluster_arn)[4]
  }  )
  tags = local.workspace["tags"]

}

Now, another stack is necessary, the IAM role to enable service account for argocd use the IAM authentication with spoke clusters. The module terraform-aws-iam/iam-eks-role allows to create the IRSA role but also is necessary create a custom policy to allow assume role in the spoke accounts. The Figure 2 depicts in depth this setup.

You can create simple stack for managing the role, or a module that supports the EKS definition and IAM role.

Figure 2. GitOps authentication summary.

So, the module is in modules/terraform-aws-irsa-eks-hub

modules/
├── terraform-aws-irsa-eks-hub
│   ├── README.md
│   ├── main.tf
│   ├── outputs.tf
│   └── variables.tf
└── terraform-aws-ssm-parameter-sotre
    ├── README.md
    ├── data.tf
    ├── main.tf
    ├── outputs.tf
    └── variables.tf

3 directories, 9 files

and the stack is:

#eks_role-terragrunt.hcl

include "root" {
  path = find_in_parent_folders("root.hcl")
  expose = true
}


dependency "eks" {
  config_path = "${get_parent_terragrunt_dir("root")}/infrastructure/containers/eks_control_plane"
  mock_outputs = {
    cluster_name = "dummy-cluster-name"
    cluster_endpoint = "dummy_cluster_endpoint"
    cluster_certificate_authority_data = "dummy_cluster_certificate_authority_data"
    cluster_version = "1.31"
    cluster_platform_version = "1.31"
    oidc_provider_arn =  "dummy_arn"
  }
  mock_outputs_merge_strategy_with_state = "shallow"
}
locals {
  # Define parameters for each workspace
  env = {
    default = {

      environment  = "control-plane"
      role_name    = "eks-role-hub"
      tags = {
        Environment = "control-plane"
        Layer       = "Networking"
      }
    }
    "dev" = {

      create = true
    }
    "prod" = {

      create = true
    }
  }
  # Merge parameters
  environment_vars = contains(keys(local.env), include.root.locals.environment.locals.workspace) ? include.root.locals.environment.locals.workspace : "default"
  workspace = merge(local.env["default"], local.env[local.environment_vars])
}


terraform {
  source = "../../../modules/terraform-aws-irsa-eks-hub"

}

inputs = {
  role_name = "${local.workspace["role_name"]}-${local.workspace["environment"]}"

  cluster_service_accounts = {
    "${dependency.eks.outputs.cluster_name}" = [
      "argocd:argocd-application-controller",
      "argocd:argo-cd-argocd-repo-server",
      "argocd:argocd-server",
    ]
  }
  tags = local.workspace["tags"]

}

Finally, the gitops_bridge stack must look like:

#eks_control_plane-terragrunt.hcl
include "root" {
  path = find_in_parent_folders("root.hcl")
  expose = true
}

include "k8s_helm_provider" {
  path = find_in_parent_folders("/common/additional_providers/provider_k8s_helm.hcl")
}

dependency "eks" {
  config_path = "${get_parent_terragrunt_dir("root")}/infrastructure/containers/eks_control_plane"
  mock_outputs = {
    cluster_name = "dummy-cluster-name"
    cluster_endpoint = "dummy_cluster_endpoint"
    cluster_certificate_authority_data = "dummy_cluster_certificate_authority_data"
    cluster_version = "1.31"
    cluster_platform_version = "1.31"
    oidc_provider_arn =  "dummy_arn"
  }
  mock_outputs_merge_strategy_with_state = "shallow"
}
dependency "eks_role" {
  config_path = "${get_parent_terragrunt_dir("root")}/infrastructure/iam/eks_role"
  mock_outputs = {
    iam_role_arn = "arn::..."
  }
  mock_outputs_merge_strategy_with_state = "shallow"

}
locals {
  # Define parameters for each workspace
  env = {
    default = {

      environment  = "control-plane"
      oss_addons = {
        enable_argo_workflows = true
        #enable_foo            = true
        # you can add any addon here, make sure to update the gitops repo with the corresponding application set
      }

      addons_metadata = merge(
        {
          addons_repo_url      = "https://github.com/gitops-bridge-dev/gitops-bridge-argocd-control-plane-template"
          addons_repo_basepath = ""
          addons_repo_path     ="bootstrap/control-plane/addons"
          addons_repo_revision = "HEAD"
        }
      )
      argocd_apps = {
        addons = file("./bootstrap/addons.yaml")
        #workloads = file("./bootstrap/workloads.yaml")
      }

      tags = {
        Environment = "control-plane"
        Layer       = "Networking"
      }
    }
    "dev" = {

      create = true
    }
    "prod" = {

      create = true
    }
  }
  # Merge parameters
  environment_vars = contains(keys(local.env), include.root.locals.environment.locals.workspace) ? include.root.locals.environment.locals.workspace : "default"
  workspace = merge(local.env["default"], local.env[local.environment_vars])
}


terraform {
  source = "tfr:///gitops-bridge-dev/gitops-bridge/helm?version=0.1.0"

}

inputs = {
  cluster_name                       = dependency.eks.outputs.cluster_name
  cluster_endpoint                   = dependency.eks.outputs.cluster_endpoint
  cluster_platform_version           = dependency.eks.outputs.cluster_platform_version
  oidc_provider_arn                  = dependency.eks.outputs.oidc_provider_arn
  cluster_certificate_authority_data = dependency.eks.outputs.cluster_certificate_authority_data

  cluster = {
    cluster_name =   dependency.eks.outputs.cluster_name
    environment  = local.workspace["environment"]
    metadata     = local.workspace["addons_metadata"]
    addons = merge(local.workspace["oss_addons"], { kubernetes_version = dependency.eks.outputs.cluster_version })

  }
  apps = local.workspace["argocd_apps"]
  argocd = {
    namespace = "argocd"
    #set = [
    #  {
    #    name  = "server.service.type"
    #    value = "LoadBalancer"
    #  }
    #]
    values = [
      yamlencode(
        {
          configs = {
            params = {
              "server.insecure" = true
            }
          }
          server = {
            "serviceAccount" = {
              annotations = {
                "eks.amazonaws.com/role-arn" = dependency.eks_role.outputs.iam_role_arn
              }

            }
            service = {
              type = "NodePort"
            }

            ingress = {
              enabled    = false
              controller = "aws"
              ingressClassName : "alb"


              aws = {
                serviceType : "NodePort"
              }

              annotations = {
                #"alb.ingress.kubernetes.io/backend-protocol" = "HTTPS"
                #"alb.ingress.kubernetes.io/ssl-redirect"                       = "443"
                #"service.beta.kubernetes.io/aws-load-balancer-type"            = "external"
                #"service.beta.kubernetes.io/aws-load-balancer-nlb-target-type" = "ip"
                #"alb.ingress.kubernetes.io/listen-ports" : "[{\"HTTPS\":443}]"
              }
            }
          }
          controller = {
            "serviceAccount" = {
              annotations = {
                "eks.amazonaws.com/role-arn" =  dependency.eks_role.outputs.iam_role_arn
              }

            }
          }
          repoServer = {
            "serviceAccount" = {
              annotations = {
                "eks.amazonaws.com/role-arn" =  dependency.eks_role.outputs.iam_role_arn
              }

            }
          }
        }
      )
    ]

  }

  tags = local.workspace["tags"]

}

Basically the main changes was the introduction to argocd map to setup the values for helm chart deployment to enable to use the IRSA role.

When you are using cross account deployment the profile that creates the secrets in hub cluster must to have access and permissions, for example in the repository the eks_control_plane stack introduce a new access entry:

#eks_control_plane-terragrunt.hcl
include "root" {
  path   = find_in_parent_folders("root.hcl")
  expose = true
}
dependency "vpc" {
  config_path = "${get_parent_terragrunt_dir("root")}/infrastructure/network/vpc"
  mock_outputs = {
    vpc_id = "vpc-04e3e1e302f8c8f06"
    public_subnets = [
      "subnet-0e4c5aedfc2101502",
      "subnet-0d5061f70b69eda14",
    ]
    private_subnets = [
      "subnet-0e4c5aedfc2101502",
      "subnet-0d5061f70b69eda14",
      "subnet-0d5061f70b69eda15",
    ]
  }
  mock_outputs_merge_strategy_with_state = "shallow"
}

locals {
  # Define parameters for each workspace
  env = {
    default = {
      create          = false
      cluster_name    = "${include.root.locals.common_vars.locals.project}-${include.root.locals.environment.locals.workspace}-hub"
      cluster_version = "1.32"

      # Optional
      cluster_endpoint_public_access = true

      # Optional: Adds the current caller identity as an administrator via cluster access entry
      enable_cluster_creator_admin_permissions = true
      access_entries = {
        #####################################################################################################################
        # Admin installation and setup for spoke accounts - Demo purpose- must be the ci Agent Role
        ####################################################################################################################
        admins_sso = {
          kubernetes_groups = []
          principal_arn     = "arn:aws:sts::123456781234:role/aws-reserved/sso.amazonaws.com/us-east-2/AWSReservedSSO_AWSAdministratorAccess_877fe9e4127a368d"
          user_name         = "arn:aws:sts::123456781234:assumed-role/AWSReservedSSO_AWSAdministratorAccess_877fe9e4127a368d/{{SessionName}}"

          policy_associations = {
            single = {
              policy_arn = "arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy"
              access_scope = {
                type = "cluster"
              }
            }
          }
        }
      }

      cluster_compute_config = {
        enabled    = true
        node_pools = ["general-purpose"]
      }

      tags = {
        Environment = "control-plane"
        Layer       = "Networking"
      }
    }
    "dev" = {

      create = true
    }
    "prod" = {

      create = true
    }
  }
  # Merge parameters
  environment_vars = contains(keys(local.env), include.root.locals.environment.locals.workspace) ? include.root.locals.environment.locals.workspace : "default"
  workspace        = merge(local.env["default"], local.env[local.environment_vars])
}


terraform {
  source = "tfr:///terraform-aws-modules/eks/aws?version=20.33.1"

}

inputs = {
  create          = local.workspace["create"]
  cluster_name    = local.workspace["cluster_name"]
  cluster_version = local.workspace["cluster_version"]

  # Optional
  cluster_endpoint_public_access = local.workspace["cluster_endpoint_public_access"]

  # Optional: Adds the current caller identity as an administrator via cluster access entry
  enable_cluster_creator_admin_permissions = local.workspace["enable_cluster_creator_admin_permissions"]

  cluster_compute_config = local.workspace["cluster_compute_config"]

  vpc_id         = dependency.vpc.outputs.vpc_id
  subnet_ids     = dependency.vpc.outputs.private_subnets
  access_entries = local.workspace["access_entries"]


  tags = {
    Environment = include.root.locals.environment.locals.workspace
    Terraform   = "true"
  }

  tags = local.workspace["tags"]

}

The final Hub infrastructure is:

Figure 3. Control Plane Infrastructure.

👇 Get the code here.

velez94 / terragrunt_aws_gitops_blueprint

Public demo for Gitops bridge using terragrunt, OpenTofu and EKS

AWS GitOps Blueprint with Terragrunt

This project provides a blueprint for implementing GitOps on AWS using Terragrunt and Argo CD. It offers a structured approach to managing infrastructure as code and deploying applications across multiple environments.

The blueprint is designed to streamline the process of setting up a GitOps workflow on AWS, leveraging Terragrunt for managing Terraform configurations and Argo CD for continuous deployment. It includes configurations for essential AWS services such as EKS (Elastic Kubernetes Service) and VPC (Virtual Private Cloud), as well as GitOps components for managing cluster addons and platform-level resources.

Key features of this blueprint include:

Modular infrastructure setup using Terragrunt
EKS cluster configuration for container orchestration
VPC network setup for secure and isolated environments
GitOps bridge for seamless integration between infrastructure and application deployments
Argo CD ApplicationSets for managing cluster addons and platform resources
Environment-specific configurations for multi-environment deployments

Repository Structure

terragrunt_aws_gitops_blueprint/
├── common/
│

…

View on GitHub

Creating spoke Cluster Infrastructure

The second step is creating the spoke cluster infrastructure. 🧑🏾‍💻

To manage a spoke cluster a single terragrunt or tofu project is created independently for each team from a template, so, infrastructure has an individual pipeline to manage it, and each team could have custom CI/CD agents, also more flexibility to add features and components to the infrastructure stacks. In some cases, you can have a single pipeline to manage the infrastructure setup and work with environment or parameters managed by the orchestration CI/CD tool. This approach is utilized when necessary to have central governance and control, but consider the changes rates, common tasks and environment setup and CI/CD workers capacity assigned to central ops.

The code is like the hub repository; however, the main difference is in the stack GitOps bridge.

Let’s watch it in depth. 🕵️‍♀️

First, a new provider configuration is necessary, the cluster hub provider, in the terragrunt_aws_gitops_spoke_blueprint/common/additional_providers:

locals {
  workspace = get_env("TF_VAR_env", "dev")
  pipeline = "false"
  hub_account_id    = "105171185823"
}
generate "k8s_helm_provider" {
  path      = "k8s_helm_provider.tf"
  if_exists = "overwrite"
  contents = <<EOF
################################################################################
# Kubernetes Access for Spoke Cluster
################################################################################

# First, define the parameter store data source
data "aws_ssm_parameter" "hub_cluster_config" {
  count = 1
  with_decryption = true
  name  = "arn:aws:ssm:us-east-2:${local.hub_account_id}:parameter/control_plane/${local.workspace}/credentials"
 #"/control_plane/${local.workspace}/credentials"  # Adjust the parameter path as needed
}

provider "kubernetes" {
  host = try(jsondecode(data.aws_ssm_parameter.hub_cluster_config[0].value).cluster_endpoint, var.cluster_endpoint)
  cluster_ca_certificate = try(base64decode(jsondecode(data.aws_ssm_parameter.hub_cluster_config[0].value).cluster_certificate_authority_data), var.cluster_certificate_authority_data)

  exec {
    api_version = "client.authentication.k8s.io/v1beta1"
    command     = "aws"
    args = [
      "eks",
      "get-token",
      "--cluster-name",
      try(jsondecode(data.aws_ssm_parameter.hub_cluster_config[0].value).cluster_name, var.cluster_name),
      "--region",
      try(jsondecode(data.aws_ssm_parameter.hub_cluster_config[0].value).cluster_region, data.aws_region.current.name),
      "--profile",
      var.profile["${local.workspace}"]["profile"]
    ]
  }
  alias = "hub"
}

EOF
}

The gitops_bridge stack is:

#eks_control_plane-terragrunt.hcl
include "root" {
  path = find_in_parent_folders("root.hcl")
  expose = true
}

include "k8s_helm_provider" {
  path = find_in_parent_folders("/common/additional_providers/provider_k8s_hub.hcl")
}

dependency "eks" {
  config_path = "${get_parent_terragrunt_dir("root")}/infrastructure/containers/eks_spoke"
  mock_outputs = {
    cluster_name = "dummy-cluster-name"
    cluster_endpoint = "dummy_cluster_endpoint"
    cluster_certificate_authority_data = "dummy_cluster_certificate_authority_data"
    cluster_version = "1.31"
    cluster_platform_version = "1.31"
    oidc_provider_arn =  "dummy_arn"
  }
  mock_outputs_merge_strategy_with_state = "shallow"
}
locals {
  # Define parameters for each workspace
  env = {
    default = {

      environment  = "control-plane"
      oss_addons = {
        enable_argo_workflows = true
        #enable_foo            = true
        # you can add any addon here, make sure to update the gitops repo with the corresponding application set
      }

      addons_metadata = merge(
        {
          addons_repo_url      = "https://github.com/gitops-bridge-dev/gitops-bridge-argocd-control-plane-template"
          addons_repo_basepath = ""
          addons_repo_path     ="bootstrap/control-plane/addons"
          addons_repo_revision = "HEAD"
        }
      )


      tags = {
        Environment = "control-plane"
        Layer       = "Networking"
      }
    }
    "dev" = {

      create = true
    }
    "prod" = {

      create = true
    }
  }
  # Merge parameters
  environment_vars = contains(keys(local.env), include.root.locals.environment.locals.workspace) ? include.root.locals.environment.locals.workspace : "default"
  workspace = merge(local.env["default"], local.env[local.environment_vars])
}


terraform {
  source = "../../../modules/terraform-aws-gitops-bridge-spoke"

}

inputs = {
  cluster_name                       = dependency.eks.outputs.cluster_name
  cluster_endpoint                   = dependency.eks.outputs.cluster_endpoint
  cluster_platform_version           = dependency.eks.outputs.cluster_platform_version
  oidc_provider_arn                  = dependency.eks.outputs.oidc_provider_arn
  cluster_certificate_authority_data = dependency.eks.outputs.cluster_certificate_authority_data
  create_kubernetes_resources = false
  cluster = {
    cluster_name =   dependency.eks.outputs.cluster_name
    environment  = local.workspace["environment"]
    metadata     = local.workspace["addons_metadata"]
    addons = merge(local.workspace["oss_addons"], { kubernetes_version = dependency.eks.outputs.cluster_version })

  }
  hub_account_id =  include.root.locals.common_vars.locals.hub_account_id

  tags = local.workspace["tags"]

}

This stacks defines the data for cluster secret in hub cluster and metadata information for addons, the local module terraform-aws-gitops-bridge-spoke creates the access entry and for enable hub access using spoke role according to Figure 2 and reuse gitops-bridge-dev/gitops-bridge/helm with parameters to deploy the secret but not the argocd installation in spoke clusters. So the infrastructure composition for spokes IaC is:

Figure 4. Infrastructure composition.

An alternative approach is to manage the external secrets Operator and secrets manager, let a comment if you want to know how do it.

Finally, after runs the spoke stacks in Argocd Hub Server you can watch the clusters and metadata information:

Figure 5. Cluster and metadata Information.

For example some addons was deployed in spoke cluster using applications Set.

Figure 6. Applications Set in spoke clusters.

In the next post, you can learn how to custom addons and add advance setups. 🦸🦸

👇 Get the code here.

velez94 / terragrunt_aws_gitops_spoke_blueprint

Infrastructure for Spoke clusters blueprint for GitOps Bridge

AWS GitOps Scale Infrastructure with EKS Spoke Clusters

A comprehensive Infrastructure as Code (IaC) solution that enables scalable GitOps deployments on AWS using EKS clusters in a hub-spoke architecture with automated infrastructure provisioning and configuration management.

This project provides a complete infrastructure setup using Terragrunt and Terraform to create and manage AWS resources including VPC networking, EKS clusters, and GitOps tooling. It implements a hub-spoke architecture where a central hub cluster manages multiple spoke clusters through GitOps practices, enabling consistent and automated application deployments at scale.

The solution includes automated VPC creation with proper network segmentation, EKS cluster provisioning with secure configurations, and integration with GitOps tools through a bridge component that enables declarative infrastructure and application management.

Repository Structure


├── common/                          # Common configuration and variable definitions
│   ├── additional_providers/        # Provider configurations for Kubernetes, Helm, etc.
│   ├── common.hcl                  # Common Terragrunt configuration
│   ├── common.tfvars               #

…

View on GitHub

Thanks for reading and sharing! 🫶

GitOps and IaC at Scale – AWS, ArgoCD, Terragrunt, and OpenTofu – Part 1

Alejandro Velez — Thu, 13 Feb 2025 17:18:53 +0000

level 400

Welcome, builders! In this blog series, we will explore the exciting world of GitOps and Infrastructure as Code (IaC) at scale, focusing on the powerful combination of ArgoCD and Open Tofu and how those tools and frameworks can live together for improving your deployments and reduce the TOIL. The main purpose is to learn some patterns and considerations for increasing efficiency, improving the automation infrastructure provisioning without loss compliance and security requirements

Scenario

Suppose that you are a Cloud Engineer or DevOps engineer, and you have a mission, deploy a cloud native infrastructure to support a modern e-commerce application that offers the possibility to any buy and sell gems. You must implement a system to allow the operations at scale, keep the governance, use IaC and consider the least manual tasks for the setup and onboarding for the developers. The cloud provider for the organization is AWS and the main infrastructure services are EKS, RDS, VPC. Don’t forget that availability, security, and cost efficiency requirements.

In the previous blogs about IaC at scale the main questions and base guidelines was answered to apply the best practices, however, when talking about Kubernetes and cloud native apps many questions emerge from the shadows and suppose al challenge for the technical skills, for example:

What are the best practices to manage the cluster configuration at scale?
How can you connect both layers IaC and GitOps?
What are the GitOps topologies?
What are the best tools?
How can you improve developer experience?

Of course, with an AI assistant you can find pretty answers for those questions and some guides. But, we are builders and here we put a real-life example based on experiences and some challenges in a productive environment.

First, some theory and good practices 😎

The deployment patterns and GitOps Topology

If you want to know more about gitops, patterns, courses and more please visit opengitops

Key points based on the requirements:
• You must have multiple environments.
• Each environment must isolate from others.
• You must manage the cluster configuration at scale.

There are tree main Gitops topologies:
• Standalone topology
• Hub-Spoke Push Model
• Hub-Spoke Push and Pull model

For this scenario, the hub-spoke push model is the best solution. This setup involves deploying GitOps tools on a central "Hub" cluster. The Hub cluster acts as the control plane for GitOps, managing deployments to various clusters dedicated to different environments or workloads.

So, how can you deploy this topology in AWS using EKS?

First, let’s get some assumptions:
• Your accounts are in the same AWS organization.
• There is already a Networking layer configured.
• ArgoCD is the gitops tool.
• Github the VSC.
• There is no CI tool predefined, in this blog series Codepipeline and previous blog posts will be use.

Security considerations:

• The IAM roles and policies for your agents must be clear and apply the least privileged principle. Sometimes the infrastructure deployment role is different from orchestration deployment role.
• Use secrets manager to share the credentials between accounts using solid resource policy or use parameters store secure string with RAM. Both must be encrypted using KMS.
• Don’t expose your cluster endpoints to the internet, if it is necessary then restrict the access by IP range.
• The security of the pipeline must be a priority, the deployment agents can assume many privileged roles, so keep in mind to avoid security breaches.

Solution overview

Open-Source projects and tools:
Open Tofu
GitOps bridge
ArgoCD

AWS Services:

EKS
KMS
RAM
Parameter Store
ECR
IAM

The following diagrams shows the solution:

1- The code is versioned in git system, usually you have tree main kind of repositories: Infrastructure as code repositories, Argocd applications Set definitions, and app or microservices repositories. Many teams keep separated the Kubernetes manifests from code applications. This is a good practice due the code application and infrastructure definitions don’t follow the same lifecycle but other factors as mono repositories or multiple repositories are key factors. Just keep in different folders 😊.

2- The CI/CD service deploys the infrastructure components: the CI/CD agents deploy the control plane cluster, create shared resources to spoke AWS accounts, and create the argocd secret for hub cluster with the metadata to allow operations and bootstrapping configurations.

The agent’s role must be part of an access entry in EKS to allow this operation.

3- The shared resources are created are common practices that use secure string in parameter store and share it with RAM is more cost effective and practical for this setup, however, you could use secrets manager with resource policies.

4- The CI/CD service deploys the spoke resources: the agent deploys the spoke cluster and other resources. Here the agent can access the Hub Kubernetes cluster to create a secret for spoke cluster with the metadata to allow operations and bootstrapping configuration.

5- ArgoCD detects new cluster metadata and prepare to deploy de addons: Once the metadata is loaded the applications are deployed in the spoke or hub cluster, to accomplish this the Argocd application controller must assume the role in the spoke account.

In ArgoCD, the limit for metadata annotations is 262,144 characters (or 256 KB)

6- The application are created: here the deployments al created into the spoken cluster and Argocd sync with the desired state.

7- The cluster are ready to workloads.

It sounds pretty, but how can integrate both layers without overhead or manual operations?

The answer is to use the GitOps bridge framework: Is a community project that aims to showcase best practices and patterns for bridging the process of creating a Kubernetes cluster to subsequently managing everything through GitOps. It focuses on using ArgoCD or FluxCD, both of which are CNN-graduated projects.

gitops-bridge-dev / gitops-bridge

GitOps Bridge

The GitOps Bridge is a community project that aims to showcase best practices and patterns for bridging the process of creating a Kubernetes cluster to subsequently managing everything through GitOps. It focuses on using ArgoCD or FluxCD, both of which are CNCF-graduated projects.

For an example template on bootstrapping ArgoCD, see the GitHub repository GitOps Control Plane.

There are many tools available for creating Kubernetes clusters. These include "roll-your-own" solutions like kubeadm, minikube, and kind, as well as cloud-managed services like Amazon EKS. The method of cluster creation should not impact GitOps compatibility; GitOps engines should work with any tool that the user chooses for cluster creation. This includes scenarios where Kubernetes is used to create other Kubernetes clusters, such as with CAPI/CAPA, Crossplane, ACK, or any tool running inside Kubernetes to deploy Kubernetes.

The GitOps Bridge becomes extremely important in the context…

View on GitHub

Hands On

Creating a control plane

So, let’s hand in to the code for building a control plane.

1- The terragrunt project is created with the following structure:

The project also can be created with OpenTofu but here is an example using terragrunt a wrapper for opentofu and a powerful tool to keep DRY and manage IaC at scale.

In the infrastructure folder you can find two major stacks: network and containers. In containers stacks the cluster for control plane is deployed using the public aws modules and gitops bridge is deployed into the account using the official terraform module.

For example, the content for eks_control_plane stack:

#eks_control_plane-terragrunt.hcl
include "root" {
  path = find_in_parent_folders("root.hcl")
  expose = true
}
dependency "vpc" {
  config_path  = "${get_parent_terragrunt_dir("root")}/infrastructure/network/vpc"
  mock_outputs = {
    vpc_id         = "vpc-04e3e1e302f8c8f06"
    public_subnets = [
      "subnet-0e4c5aedfc2101502",
      "subnet-0d5061f70b69eda14",
    ]
     private_subnets                             = [
     "subnet-0e4c5aedfc2101502",
      "subnet-0d5061f70b69eda14",
      "subnet-0d5061f70b69eda15",
     ]
  }
  mock_outputs_merge_strategy_with_state = "shallow"
}

locals {
  # Define parameters for each workspace
  env = {
    default = {
      create       = false
      cluster_name = "${include.root.locals.common_vars.locals.project}-${include.root.locals.environment.locals.workspace}-hub"
      cluster_version = "1.31"

      # Optional
      cluster_endpoint_public_access = true

      # Optional: Adds the current caller identity as an administrator via cluster access entry
      enable_cluster_creator_admin_permissions = true

      cluster_compute_config = {
        enabled = true
        node_pools = ["general-purpose"]
      }

      tags = {
        Environment = "control-plane"
        Layer       = "Networking"
      }
    }
    "dev" = {

      create = true
    }
    "prod" = {

      create = true
    }
  }
  # Merge parameters
  environment_vars = contains(keys(local.env), include.root.locals.environment.locals.workspace) ?  include.root.locals.environment.locals.workspace : "default"
  workspace = merge(local.env["default"], local.env[local.environment_vars])
}


terraform {
  source = "tfr:///terraform-aws-modules/eks/aws?version=20.33.1"

}

inputs = {
  create= local.workspace["create"]
  cluster_name = local.workspace["cluster_name"]
  cluster_version = local.workspace["cluster_version"]

  # Optional
  cluster_endpoint_public_access = local.workspace["cluster_endpoint_public_access"]

  # Optional: Adds the current caller identity as an administrator via cluster access entry
  enable_cluster_creator_admin_permissions = local.workspace["enable_cluster_creator_admin_permissions"]

  cluster_compute_config = local.workspace["cluster_compute_config"]

  vpc_id = dependency.vpc.outputs.vpc_id
  subnet_ids = dependency.vpc.outputs.private_subnets

  tags = {
    Environment = include.root.locals.environment.locals.workspace
    Terraform   = "true"
  }

  tags = local.workspace["tags"]

}

Watch that the main properties for this stack are the cluster version and in this case the auto mode for EKS is enabled.

In the next blogs you can learn more about this!

For other hand, for gitops_bridge stack:

include "root" {
  path = find_in_parent_folders("root.hcl")
  expose = true
}


include "k8s_helm_provider" {
  path = find_in_parent_folders("/common/additional_providers/provider_k8s_helm.hcl")
}

dependency "eks" {
  config_path = "${get_parent_terragrunt_dir("root")}/infrastructure/containers/eks_control_plane"
  mock_outputs = {
    cluster_name = "dummy-cluster-name"
    cluster_endpoint = "dummy_cluster_endpoint"
    cluster_certificate_authority_data = "dummy_cluster_certificate_authority_data"
    cluster_version = "1.31"
  }
  mock_outputs_merge_strategy_with_state = "shallow"
}

locals {
  # Define parameters for each workspace
  env = {
    default = {

      environment  = "control-plane"
      oss_addons = {
        enable_argo_workflows = true
        #enable_foo            = true
        # you can add any addon here, make sure to update the gitops repo with the corresponding application set
      }

      addons_metadata = merge(
        {
          addons_repo_url      = "https://github.com/gitops-bridge-dev/gitops-bridge-argocd-control-plane-template"
          addons_repo_basepath = ""
          addons_repo_path     ="bootstrap/control-plane/addons"
          addons_repo_revision = "HEAD"
        }
      )
      argocd_apps = {
        addons = file("./bootstrap/addons.yaml")
        #workloads = file("./bootstrap/workloads.yaml")
      }

      tags = {
        Environment = "control-plane"
        Layer       = "Networking"
      }
    }
    "dev" = {

      create = true
    }
    "prod" = {

      create = true
    }
  }
  # Merge parameters
  environment_vars = contains(keys(local.env), include.root.locals.environment.locals.workspace) ? include.root.locals.environment.locals.workspace : "default"
  workspace = merge(local.env["default"], local.env[local.environment_vars])
}


terraform {
  source = "tfr:///gitops-bridge-dev/gitops-bridge/helm?version=0.1.0"

}

inputs = {
  cluster_name                       = dependency.eks.outputs.cluster_name
  cluster_endpoint                   = dependency.eks.outputs.cluster_endpoint
  cluster_platform_version           = dependency.eks.outputs.cluster_platform_version
  oidc_provider_arn                  = dependency.eks.outputs.oidc_provider_arn
  cluster_certificate_authority_data = dependency.eks.outputs.cluster_certificate_authority_data

  cluster = {
    cluster_name =   dependency.eks.outputs.cluster_name
    environment  = local.workspace["environment"]
    metadata     = local.workspace["addons_metadata"]
    addons = merge(local.workspace["oss_addons"], { kubernetes_version = dependency.eks.outputs.cluster_version })

  }
  apps = local.workspace["argocd_apps"]

  tags = local.workspace["tags"]

}

Here the addonsare enabled or disable using flags, in oss_addonsmap also you can and the ArgoCD application Set to bootstrap the cluster.

2- Create the application set for addons:

kind: ApplicationSet
metadata:
  name: cluster-addons
  namespace: argocd
spec:
  syncPolicy:
    preserveResourcesOnDeletion: true
  generators:
  - clusters:
      selector:
        matchLabels:
          environment: control-plane
  template:
    metadata:
      name: 'cluster-addons'
    spec:
      project: default
      source:
        repoURL: '{{metadata.annotations.addons_repo_url}}'
        path: '{{metadata.annotations.addons_repo_basepath}}{{metadata.annotations.addons_repo_path}}'
        targetRevision: '{{metadata.annotations.addons_repo_revision}}'
        directory:
          recurse: true
      destination:
        namespace: 'argocd'
        name: '{{name}}'
      syncPolicy:
        automated:
          allowEmpty: true

The key point here is the generators, where the cluster selector is just deploy the application in the clusters that have the label environment equal to control plane.
The source for de applications is the addons_repo_url defined in gitops bridge stack terragrunt_aws_gitops_blueprint/infrastructure/containers/gitops_bridge/terragrunt.hcl.

3- Connect to the cluster using a k8s client.
Setup the kube context using aws cli:

aws eks update-kubeconfig --name my-cluster --region us-east-1 --alias my-cluster-alias

For example:

aws eks update-kubeconfig --name gitops-scale-dev-hub --region us-east-2 --alias labvel-devsecops-hub --profile labvel-dev

Run:

k9s --context labvel-devsecops-hub

for interactive mode or just find the default ArgoCD server admin secret and run kubectl to create a forwarding to connect with the argocd server.

kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" --context labvel-devsecops-hub | base64 -d; echo

kubectl port-forward svc/argo-cd-argocd-server -n argocd 8080:443 --context labvel-devsecops-hub

Finally through the web browser you can find the argo apps:

The application Set:

And the cluster metadata:

In the next part you can learn more about enable and disable addons and apps and go deeper using AWS controllers.

You can find the public code here:

velez94 / terragrunt_aws_gitops_blueprint

Public demo for Gitops bridge using terragrunt, OpenTofu and EKS

AWS GitOps Blueprint with Terragrunt

Key features of this blueprint include:

Modular infrastructure setup using Terragrunt
EKS cluster configuration for container orchestration
VPC network setup for secure and isolated environments
GitOps bridge for seamless integration between infrastructure and application deployments
Argo CD ApplicationSets for managing cluster addons and platform resources
Environment-specific configurations for multi-environment deployments

Repository Structure

terragrunt_aws_gitops_blueprint/
├── common/
│

…

View on GitHub

Thanks for reading and sharing! ☺️⭐

DevSecOps with AWS- IaC at scale - Building your own platform – Part 3 - Pipeline as a Service

Alejandro Velez — Sat, 30 Nov 2024 23:30:51 +0000

Level 300

In the previous post for this series,DevSecOps with AWS- IaC at scale - Building your own platform - Part 1 and DevSecOps with AWS- IaC at scale - Building your own platform - Part 2 - CI for IaC you could learn more about creating the CI pipeline, key questions and practices to enable the self-service capabilities to reuse this pipeline. In the following example, you will learn a use case to allow self-services capabilities using the pipeline of pipeline approach and combine the service Catalog capabilities to get a Pipeline as a Service.

So, let’s create as Platform engineer. But first a little overview about the solution:

Solution Overview

Now review the solution exposed in the previous blog post. And focus on creating a SaaS model to deploy your standard pipelines in other DevSecOps accounts. Using CDK Pipelines for this approach. In the future post you can enable the self-service capabilities for now just focusing on the platform’s bases.

You need to follow the next steps:
1- Construct pipeline of pipelines using CDK Pipelines.
2- Test the pipeline of pipelines to create a pipeline for deployment in sandbox account.
3- Approve changes to pass to production environments.
4- Enable self service capabilities to add new project to the pipelines of pipelines.
5- Out the box, create self service portal according to IDP (Internal Developer Portal).

Requirements

• cdk >= 2.167.1
• AWS CLI >= 2.7.0
• Python >= 3.10.4

AWS Services

AWS Cloud Development Kit (CDK): is an open-source software development framework to define your cloud application resources using familiar programming languages.
AWS Identity and Access Management (IAM): Securely manage identities and access to AWS services and resources.
AWS IAM Identity Center (Successor to AWS Single Sign-On): helps you securely create or connect your workforce identities and manage their access centrally across AWS accounts and applications.
AWS CodeBuild: fully managed continuous integration service that compiles source code, runs tests, and produces software packages that are ready to deploy.
AWS CodePipeline: fully managed continuous delivery service that helps you automate your release pipelines for fast and reliable application and infrastructure updates.
AWS Key Management Service (AWS KMS): lets you create, manage, and control cryptographic keys across your applications and more than 100 AWS services.
AWS CloudFormation: Speed up cloud provisioning with infrastructure as code.
AWS Resource Access Manager: Simply and securely share your AWS resources across multiple accounts.

Hands On ☺️

First create a CDK pipelines Stack and deployment stage:

from aws_cdk import (
    Stage,
    Environment,
    Tags
)
from constructs import Construct

from ...cdkv2_dev_sec_ops_ia_c_terraform_stack import Cdkv2DevSecOpsIaCTerraformStack


class PipelineStageProd(Stage):

    def __init__(self, scope: Construct, id: str, props: dict = None,
                 compliance_buildef: dict = None,
                 terra_plan_buildef: dict = None,
                 kics_buildef: dict = None,
                 tfsec_buildef: dict = None,
                 checkov_buildef: dict = None,
                 testing_buildef: dict = None,
                 deployment_buildef: dict = None,
                 env_devsecops_account: Environment = None,
                 environments: dict = None,
                 tags:dict = None, **kwargs):
        super().__init__(scope, id, **kwargs)

        core_stack = Cdkv2DevSecOpsIaCTerraformStack(self, f"CoreDevSecOpsIaCTerraformStack-{props['project_name']}",
                                                     stack_name=f"CoreDevSecOpsIaCTerraformStack-{props['project_name']}",

                                                     props=props,
                                                     compliance_buildef=compliance_buildef,
                                                     tfsec_buildef=tfsec_buildef,
                                                     checkov_buildef=checkov_buildef,
                                                     testing_buildef=testing_buildef,
                                                     terra_plan_buildef=terra_plan_buildef,
                                                     kics_buildef=kics_buildef,
                                                     deployment_buildef=deployment_buildef,
                                                     env_devsecops_account=env_devsecops_account,
                                                     environments=environments

                                                     )

        for t in tags.keys():
            Tags.of(core_stack).add(key=t, value=tags[t])

Now, from main stack add the stages based on the project properties in yaml files, here a common question for manage parallel deployments and projects with this approach.

How can we deploy many stacks parallel when we have many projects?

The answer is using the CDK Pipelines waves capability, this capability is useful in several scenarios:

Parallel Deployments:
• Deploy multiple stages simultaneously to different regions or accounts.
• Reduce overall pipeline execution time by running independent deployments in parallel.
• Deploy the same application to multiple environments concurrently.
Deployment Organization:
• Group related deployments logically.
• Create deployment waves for different environments (dev, test, prod).
• Organize deployments by business units or applications.
Controlled Progressive Rollouts:
• Deploy to a subset of regions/accounts first.
• Implement progressive deployment patterns.
• Add validation steps between waves for safety.
Resource Optimization:
• Control concurrent deployments to manage resource usage.
• Balance deployment speed with system load.
• Optimize pipeline execution costs.
Dependency Management:
• Group stages that can run independently.
• Separate dependent deployments into sequential waves.
• Manage complex deployment orchestration.

While stages within a wave run in parallel, the waves themselves execute sequentially. This allows for controlled progression through your deployment pipeline while maximizing efficiency where parallel execution is beneficial.

Second, test the stack with the parameters, in this case all definitions in a folder project configs are loaded, and one pipeline is created for each parameters set (team or project definitions).

Each project properties are loaded with a python function and create a custom class to abstract the project configurations.

import os
from dataclasses import dataclass
from typing import Dict, Any, Optional
from aws_cdk import Environment
from .utils import load_yamls, load_yamls_to_dict, load_tags, find_yaml_files

@dataclass
class ProjectConfiguration:
    """Class to manage project configuration and environments"""

    def __init__(self, props_path: str, dirname: Optional[str] = None):
        """
        Initialize ProjectConfiguration

        Args:
            props_path (str): Path to the properties file
            dirname (str, optional): Base directory path. If None, uses the directory of this class
        """
        self.dirname = dirname if dirname is not None else os.path.dirname(__file__)
        self.props_path = props_path
        self.props = self._load_properties()
        self.environments = {}
        self.build_definitions = {}

        self._setup_environments()
        self._load_tags()
        self._load_build_definitions()

    def _load_properties(self) -> Dict[str, Any]:
        """Load main properties from YAML file"""
        full_path = os.path.join(self.props_path)
        return (load_yamls(full_path))[0]

    def _setup_environments(self) -> None:
        """Setup environment configurations"""
        self.devsecops_env = Environment(
            account=self.props['account_devsecops'],
            region=self.props['region_devsecops']
        )

        self.props["def_environments"] = {}
        for env_config in self.props["environments"]:
            self._process_environment(env_config)

        self.props["environments"] = self.environments

    def _process_environment(self, env_config: Dict[str, Any]) -> None:
        """Process individual environment configuration"""
        env_name = env_config["environment"]

        # Create CDK Environment
        self.environments[env_name] = Environment(
            account=env_config['deployment_account'],
            region=env_config['deployment_region']
        )

        # Basic environment configuration
        self.props["def_environments"][env_name] = {
            "deployment_region": env_config["deployment_region"],
            "deployment_account": env_config["deployment_account"],
            "enable": env_config["enable"],
            "manual_approval": env_config["manual_approval"],
            "peer_review_approval": env_config["peer_review_approval"]
        }

        # Test environment specific configuration
        if env_name == "test":
            self.props["def_environments"][env_name].update({
                "automate_testing": env_config["automate_testing"],
                "test_workspace": env_config["test_workspace"]
            })

        # Optional partner review email
        if "partner_review_email" in env_config:
            self.props["def_environments"][env_name]["partner_review_email"] = env_config["partner_review_email"]

    def _load_tags(self) -> None:
        """Load tags from YAML file"""
        tags = (load_yamls(os.path.join(self.props_path)))[1]['tags']
        self.props["tags"] = load_tags(tags)

    def _load_build_definitions(self) -> None:
        """Load all build definitions"""
        definition_mappings = {
            'compliance': 'path_def_compliance',
            'terra_plan': 'path_def_create_artifacts',
            'tfsec': 'path_def_tfsec',
            'checkov': 'path_def_checkov',
            'kics': 'path_def_kics',
            'testing': 'path_def_testing',
            'pipeline_cd': 'path_def_deploy'
        }

        for key, path_key in definition_mappings.items():
            self.build_definitions[key] = load_yamls_to_dict(
                os.path.join(self.dirname, self.props[path_key])
            )

    def get_environment(self, env_name: str) -> Environment:
        """Get specific environment configuration"""
        return self.environments.get(env_name)

    def get_build_definition(self, definition_type: str) -> Dict[str, Any]:
        """Get specific build definition"""
        return self.build_definitions.get(definition_type, {})


# Example usage with different configuration files
def get_project_configs():
    print("Loading project configs ...")
    dirname = os.path.dirname("./project_configs/environment_options/")
    dirname_pipes = os.path.dirname("./project_configs/pipeline_definitions")
    projects = find_yaml_files(dirname, "environment_options_terragrunt")

    project_configs= []
    for p in projects:
    # Create instances for different environments or configurations
        conf=ProjectConfiguration(dirname= dirname_pipes,props_path=p)
        project_configs.append(conf)
    return project_configs

Third, test in sandbox and add the appropriate approval steps.

Here code fragment for this deployment, here two waves are created according to reference diagram, the first deploy a sandbox template and after pass to production environments.

from aws_cdk import (
    Stack,
    Environment,
    Aws,
    aws_codecommit as codecommit,
    pipelines
)
from constructs import Construct
from .load_project_configs import get_project_configs
from .stages.deploy_pipeline_stack import PipelineStageProd


class Cdkv2DSOTerraformPipelineStack(Stack):

    def __init__(self, scope: Construct, construct_id: str, props: dict = None,

                 **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)


        repo = props['infra_repository']["name"]
        synth = pipelines.ShellStep(
            "Synth",



      input=pipelines.CodePipelineSource.connection(repo_string=f"{props['infra_repository']['owner']}/{repo}",
                                                          code_build_clone_output=True,

                                                          branch=props['infra_repository']['branch'],
                                                          connection_arn=f"arn:aws:codestar-connections:{Aws.REGION}:{Aws.ACCOUNT_ID}:connection/{props['infra_repository']['connection_id']}"
                                                          ),

            commands=[
                "npm install -g aws-cdk",  # Installs the cdk cli on Codebuild
                "pip install -r requirements.txt",
                "pip install checkov",
                # Instructs Codebuild to install required packages
                "npx cdk synth",
            ]
        )

        pipeline = pipelines.CodePipeline(self, f"CDKPipeline-{props['project_name']}",
                                          self_mutation=True,
                                          cross_account_keys=True,
                                          synth=synth,
                                          pipeline_name=repo,

                                          )

        sandbox_wave = pipeline.add_wave("DeploySandbox")
        prod_wave = pipeline.add_wave("DeployProd")


        # add manual approval for prod wave
        sandbox_wave.add_post(pipelines.ManualApprovalStep("PromoteToProd",
                                                           comment="Ready to apply to prod deployments"))
        projects = get_project_configs()
        for project in projects:
            project_name = project.props.get("project_name")
            print(project_name)

            stage_id = f"Deploy-{project_name}"
            # create wave
            deploy_stage = PipelineStageProd(self, stage_id, stage_name=stage_id,
                                             props=project.props,
                                             compliance_buildef=project.get_build_definition("compliance"),
                                             tfsec_buildef=project.get_build_definition("tfsec"),
                                             checkov_buildef=project.get_build_definition("checkov"),
                                             testing_buildef=project.get_build_definition("testing"),
                                             terra_plan_buildef=project.get_build_definition("terra_plan"),
                                             kics_buildef=project.get_build_definition("kics"),
                                             deployment_buildef=project.get_build_definition("pipeline_cd"),
                                             env=project.devsecops_env,  # env_client_prd_account,
                                             env_devsecops_account=project.devsecops_env,
                                             environments=project.environments,
                                             tags=project.props.get("tags")
                                             )  # env=env_client_prd_account)
            if project_name.endswith("sandbox"):
                sandbox_wave.add_stage(
                    deploy_stage,

                )
            else:
                prod_wave.add_stage(
                    deploy_stage
                )

You can modify it according to your requirements.

The AWS console view is something like this:

Finally, you are ready to deploy more projects, in this case the project structure allow put a file for each project in the path and the cdk app automatically extend and create new pipelines:


tree project_configs/environment_options/
project_configs/environment_options/
├── environment_options_advanceiac_v2.yaml
├── environment_options_ecs_fargate_pattern.yaml
├── environment_options_project_1.yaml
├── environment_options_project_2.yaml
├── environment_options_project_3_networking.yaml
├── environment_options_project_4_shared.yaml
├── environment_options_terragrunt_blueprint.yaml
├── environment_options_terragrunt_sandbox.yaml
└── environment_options_another_project.yaml

1 directory, 9 files

Thanks for reading and sharing. If you want to know more and get the code please follow me and at the end of this series I will share the final project version as opensource project.

Continuous Delivery applied to Authorization with IAM Identity Center and AWS IAM Access Analyzer – Part 2

Alejandro Velez — Sun, 22 Sep 2024 01:01:16 +0000

level: 300

According to part 1 let’s continue with the pipeline creation. In this scenario, CDK pipelines will be the preferred tool to make this possible. But in the third part, you can explore this with Terraform and Codecatalyst project.

Solution Overview

Requirements

validate-aws-policies
Python >= 3.10.4
CDK >= 2.158.0
cdk_nag >= 2.28.195

AWS Services

AWS IAM Access Analyzer: AWS Identity and Access Management (IAM) is a web service that helps you securely control access to AWS resources. With IAM, you can manage permissions that control which AWS resources users can access. You use IAM to control who is authenticated (signed in) and authorized (has permissions) to use resources. IAM provides the infrastructure necessary to control authentication and authorization for your AWS accounts.
AWS Cloud Development Kit (CDK): is an open-source software development framework to define your cloud application resources using familiar programming languages.
AWS Identity and Access Management (IAM): Securely manage identities and access to AWS services and resources.
AWS IAM Identity Center (Successor to AWS Single Sign-On): helps you securely create or connect your workforce identities and manage their access centrally across AWS accounts and applications.
AWS CodeBuild: fully managed continuous integration service that compiles source code, runs tests, and produces software packages that are ready to deploy.
AWS CodePipeline: fully managed continuous delivery service that helps you automate your release pipelines for fast and reliable application and infrastructure updates.
AWS Key Management Service (AWS KMS): lets you create, manage, and control cryptographic keys across your applications and more than 100 AWS services.
AWS CloudFormation: Speed up cloud provisioning with infrastructure as code as code.
AWS Lambda: A serverless compute service that lets you run code without provisioning or managing servers, build workload-based cluster scaling logic, maintain event integrations, or manage runtimes.
AWS Chatbot: Monitor, operate, and troubleshoot your AWS resources with interactive ChatOps.

The Figure 1 depicts the solution architecture according to best practices:

Figure 1. Continuous authorization using AWS Developer Tools

1- The IaC is hosted in github private repository.
2- The first stage for CDK pipelines synth and apply self mutation.
3- The policies are scanned by validate_aws_policies tool and push the reports into S3 bucket.
4- After the DevSecOps Adm, SecOps Engineer review the findings accept or reject the changes.
5- The permissions sets changes are provisioned in both accounts. You can modify to only apply the changes to one account but keep in mind that in this case the same team manage both accounts.

Keep in mind:

The IAM Identity Center could be delegated administration to an administrator account. https://docs.aws.amazon.com/singlesignon/latest/userguide/delegated-admin.html
The delegated administrator can’t modify or alter permissions set provisioned in the management account.
You must manage IAM Identity Center for both account instances. (Delegated and management Account).

Hands On

It’s time to create some code. 😃

First, delegate the IAM Identity Center administration using the AWS console or through the API.

Sign in to the AWS Management Console using the credentials of your management account in AWS Organizations. Management account credentials are required to run the RegisterDelegatedAdministrator API.
Select the Region where IAM Identity Center is enabled, and then open the IAM Identity Center console.
Choose Settings, and then select the Management tab.
In the Delegated administrator section, choose Register account.
On the Register delegated administrator page, select the AWS account you want to register, and then choose Register account.

Now, parametrize the project properties according to template as a follow:

Get the values from IAM Identity Center settings, you need the instance ID and the instance ARN.

Figure 2. SSO instance information.

Get the group’s principal ID from the console or run a tool like reverse_diagrams to get this information. For example:

$ reverse_diagrams -o -i --profile labvel-master --region us-east-2

The json output for the reverse_diagrams cli in file diagrams/json/groups.json:

...
{
    "AWSLogArchiveAdmins": {
        "group_id": "9a672b3314-c481fbee-8062-432a-8b87-xxxx36b763a8",
        "group_name": "AWSLogArchiveAdmins",
        "members": []
    }, 
...

Now, parametrize the project properties according to your environment, create the permission set block in project properties according to the manage or custom policies for permissions set, for example:

…
permissions_set:
  - name: 'AWSLogArchiveAdmins'
    description: 'Permissions Set for DevSecOps Admins'
    policies_file: 'policies/policy_allow_all_access_dev_sandbox_users.json'
    managed_policies: [ 'arn:aws:iam::aws:policy/AdministratorAccess' ]
    session_duration: '8'
    assing_to:
      - name: 'Grp-AWS-DevSecOps-Productos'
        principal_id: "9a672b3314-c481fbee-8062-432a-8b87-xxxx36b763a8"
        principal_type: "GROUP"
        target_ids:
          - "123462754109"

        target_type: "AWS_ACCOUNT"

…

For deploying, run:


cdk deploy -e ManageIAMIdentityCenterPipelineStack --profile labvel-devsecops

⚠️ The Github connection is disable by default, you must enable through the console.

Figure 3. Github connection in AWS Console.

Finally push the changes and wait for approving the pipeline.

Here an example for failed execution:

Figure 4. Failed execution in Code Pipeline.

Figure 5. Execution Notification in Microsoft Teams.

The Figure 4. Depicts the failed execution due case a custom policy is malformed and has a wildcard without like operator.

After clean up the repository and pass the correct policies the Figure 6. depicts the outputs in Microsoft Teams and the pipeline steps.

Figure 6. Successful execution.

You can find the example code for this pipeline.

velez94 / cdkv2_manage_identity_center_template

Template example for manage identity center authorization using aws cdk and validate_aws_policies cli

Welcome to Manage Identity Center Authorization with AWS Developer tools.

This is a project for CDK development with Python.

Architecture Diagram

Please read: Continuous Delivery applied to Authorization with IAM Identity Center and AWS IAM Access Analyzer – Part 2

The CDK instructions

The cdk.json file tells the CDK Toolkit how to execute your app.

This project is set up like a standard Python project. The initialization process also creates a virtualenv within this project, stored under the .venv directory. To create the virtualenv it assumes that there is a python3 (or python for Windows) executable in your path with access to the venv package. If for any reason the automatic creation of the virtualenv fails, you can create the virtualenv manually.

To manually create a virtualenv on MacOS and Linux:

$ python3 -m venv .venv

After the init process completes and the virtualenv is created, you can use the…

View on GitHub

Thanks for reading and sharing. 💻 👽

Continuous Delivery applied to Authorization with IAM Identity Center and AWS IAM Access Analyzer – Part 1

Alejandro Velez — Sun, 04 Aug 2024 21:14:55 +0000

Level 200

A common task for security engineers is to grant the correct permissions set around the AWS environments applying the least privilege, keeping a central review access, continuous refinement, continuous policy validation. AWS IAM access analyzer provides these capabilities and allows you automate the process.
In this series you can learn how to apply the best practices and automate the process using IAM Access Analyzer and Amazon Boto3 SDK to automate the validation process with AWS Developer tools.

Use case

Imagine that you are a security engineer and must validate the IAM policies documents for IAM identity policies and Inline policies for permissions set. At the beginning a lot of these policies are defined in json format in source code and you must include the validation steps into the CD pipeline to manage the authorization as code.

Solution Overview

Requirements

validate-aws-policies
Python >= 3.10.4

AWS Services

AWS IAM Access Analyzer: AWS Identity and Access Management (IAM) is a web service that helps you securely control access to AWS resources. With IAM, you can manage permissions that control which AWS resources users can access. You use IAM to control who is authenticated (signed in) and authorized (has permissions) to use resources. IAM provides the infrastructure necessary to control authentication and authorization for your AWS accounts.
AWS SDK for Python (Boto3): Boto3 makes it easy to integrate your Python application, library, or script with AWS services including Amazon S3, Amazon EC2, Amazon DynamoDB, and more.

Hands On

First, explore the command line tool validate-aws-policies, this allows you to validate a set of policies defined into a folder in simple json format. This tool also allows to create the html report or pdf and also publish the results into a bucket in zip format.

validate-aws-policies -h
usage: validate-aws-policies [-h] [-c] [-u UPLOAD_REPORT] [-b] [-d DIRECTORY_POLICIES_PATH] [-p PROFILE] [-z] [-cp] [-v]

options:
  -h, --help            show this help message and exit
  -c, --ci              Run into pipeline if it's present
  -u UPLOAD_REPORT, --upload_report UPLOAD_REPORT
                        Upload reports to s3 bucket
  -b, --bucket_name     Use this flag for setting the bucket tool if --upload_report is present.
  -d DIRECTORY_POLICIES_PATH, --directory_policies_path DIRECTORY_POLICIES_PATH
                        Path where Policies are defined in json format
  -p PROFILE, --profile PROFILE
                        AWS cli profile for Access Analyzer Api
  -z, --zip_reports     Set in True if you want to create a zip file for reports
  -cp, --create_pdf_reports
                        Set it if you want to create a pdf report, this need wkhtmltopdf file for reports
  -v, --version         Print the package version

For example:

You must have a session with an AWS Profile, for this example the profile name is labvel-devsecops.

Finally, you can watch the html report in your browser or in the terminal output.

The report is created in the same path where you run the command with the date and in html report by default for example: AccessAnalyzerReport_2024-08-04 13:12:07.285648.html.

You can find the source code here:

velez94 / validate-aws-policies

Python CLI to validate aws policies using boto3 and Access Analyzer API

Validate AWS policies

This package scan AWS SCP policies and create report in HTML and PDF format.

Pre-Requirements

Setup AWS Cli profile for interacting with IAM access analyzer API using IAM or SSO credentials.

Requirements

python >= 3.8

Install

$ pip install --upgrade validate-aws-policies

From AWS CodeArtifacts repository

You must have a user into AWS account Sophos Organization, it could be for projects, products, or IT internal Organizations. Before create AWS CLI profile using AWS IAM Identity Center (SSO) or IAM.

Configure your pip cli for download package from private CodeArtifacts repository


$ aws codeartifact login --tool pip --repository <repository_name> --domain <domain> --domain-owner <123456789012>  --profile <profile_name> --region <repository_region>
$ pip install --upgrade validate-aws-policies

From Azure Artifacts repository

Project setup

Ensure you have installed the latest version of the Azure Artifacts keyring from the "Get the tools" menu.

If you don't already have one, create a virtualenv using these instructions from…

View on GitHub

Thanks for reading and sharing!! 🤓🤓