Forem: DevOps4Me Global

Failing the DevOps Audit? Here's What Every Secure Pipeline Needs in 2025

DevOps4Me Global — Fri, 25 Apr 2025 14:22:20 +0000

"Last month, I reviewed a FinTech's CI pipeline. Everything looked smooth… until the audit flagged 37 license violations on Open-Source Software (OSS), 2 hardcoded secrets, and no SBOM at all. Sound familiar?"

It's 2025, and DevOps is everywhere - but so are audit failures. You're exposed if your team is shipping code fast but hasn't prepared for SBOM governance, CI/CD security, and AI-generated code traceability. As someone helping teams across MedTech, FinTech, and Industrial Software, here's what I've seen inside real pipelines - and why they're failing.

The 5 Most Common Compliance Pitfalls in DevOps

1. No SBOM Strategy
Most teams still don't generate a proper Software Bill of Materials (SBOM), or worse, they rely on automated SCA tools but never review the output.
Fix: Adopt SBOM tools like BlackDuck, Syft, or CycloneDX, and automate export in your build.

2. Secrets in Repos or CI Variables
I've reviewed pipelines with unencrypted secrets exposed in env variables or YAML files - a ticking time bomb.
Fix: Use Azure Key Vault, GitHub Secrets, or HashiCorp Vault with strict access policies.

3. No Policy Gates or Security Gating
CI/CD runs green even when critical vulnerabilities exist. Why? No security gate policies.
Fix: Integrate gating with Coverity, BlackDuck, Trivy, or custom rules to break builds.

4. Lack of AI Code Tracking
With the rise of AI code tools, teams can't distinguish AI-generated vs. developer-written code. This creates audit headaches under the EU CRA.
Fix: Implement tagging policies or track source commits with specific AI markers + governance checklists.

5. No Mapped Compliance Framework
Teams follow "best practices" but can't prove alignment with IEC 62443, CRA, or FDA pre-submission standards.
Fix: Build a mapping checklist showing each security control met in your pipeline tooling.
What a Secure, Audit-Ready Pipeline Looks Like
Below is a high-level view of a compliant CI/CD pipeline. From SBOM generation to secrets management and security gates - each layer aligns with audit requirements:

Want to Fix Your Pipeline Before the Auditor Does?
I've helped DevOps and Cybersecurity teams design fast audit-ready pipelines.

You can:
Download my Free CI/CD YAML
Audit Checklist
Book a 30-minute free consult (limited slots)

📩 *Get the eBook DevOps Unlocked *→https://najibradzuan.gumroad.com/l/devopsunlocked
🔗 Follow more DevSecOps and Cybersecurity posts at Dev.to or Medium

Najib Radzuan is a DevSecOps Security Architect working with teams globally on Regulatory Compliance, IEC 62443, EU CRA, and DevSecOps transformation.

Leadership in DevOps: Applying the PATH-C Framework for High-Performing Teams

DevOps4Me Global — Mon, 10 Feb 2025 06:01:47 +0000

DevOps isn’t just about automation and pipelines—it’s about people. Effective leadership is the cornerstone of high-performing DevOps teams. But how can leaders drive focus, accountability, and collaboration without micromanaging or slowing down innovation?

Enter PATH-C: A Leadership Framework for DevOps. 💡

What is PATH-C?

PATH-C is a structured approach to improving team productivity, decision-making, and collaboration. It consists of five key principles:

✅ Prioritization – Focus on what matters, avoiding distractions.
✅ Accountability – Ensure every team member owns their work.
✅ Transparency – Keep processes, decisions, and progress visible.
✅ Habits – Build sustainable routines that support efficiency.
✅ Collaboration – Foster teamwork and knowledge sharing.

🔥 Applying PATH-C in Leadership
🚀 1. Prioritization:
Leaders must help teams cut through the noise and focus on critical objectives. Use quarterly OKRs, sprint-level goals, and backlog refinement to keep priorities clear.

🚀 2. Accountability:
Ownership drives performance. Assign clear responsibilities, track commitments, and use sprint retrospectives to reinforce accountability.

🚀 3. Transparency:
Open dashboards, shared status updates, and clear documentation ensure that work isn’t hidden in silos. DevOps thrives when teams see the big picture.

🚀 4. Habits:
Good leaders establish habits that sustain productivity. Daily stand-ups, regular code reviews, and structured learning sessions keep teams aligned and growing.

🚀 5. Collaboration:
Cross-functional teamwork is essential. Dev, Ops, and Security must work together seamlessly. Promote pair programming, DevSecOps principles, and knowledge sharing across teams.

🎯 Real-World Impact
Companies that implement PATH-C experience:
✅ Faster deployments and reduced bottlenecks 🚀
✅ Higher team engagement and job satisfaction 😊
✅ Better cross-team collaboration and reduced silos 🤝
✅ Stronger security and reliability with built-in accountability 🔐

📖 Want to Dive Deeper?
Learn how PATH-C can transform your leadership style and DevOps team performance. Get "DevOps Unlocked: Mastering Productivity with the PATH-C Framework" today!

🔗 Order now → https://a.co/d/4fWAUoq

Let’s unlock productivity, leadership, and innovation in DevOps together! 💪🔥

DevOps #Leadership #Productivity #PATHC #Agile #Engineering #Automation #Collaboration #ContinuousImprovement #DevSecOps #CI/CD

DevOps Unlocked: The Ultimate Guide to Mastering Productivity with the PATH-C Framework

DevOps4Me Global — Wed, 05 Feb 2025 14:45:10 +0000

🚀 Are you a DevOps professional struggling to keep up with the ever-increasing demands of automation, security, and efficiency? Do you feel overwhelmed by the complexities of modern software development and delivery?

You’re not alone.

In today’s fast-paced DevOps landscape, productivity isn’t just about getting things done — it’s about getting the right things done efficiently, securely, and at scale. I wrote “DevOps Unlocked: Mastering Productivity with the PATH-C Framework” — to help DevOps engineers, managers, and teams optimize workflows, eliminate bottlenecks, and build sustainable productivity habits that drive long-term success.

💡 Get your copy now on Amazon! 👉 DevOps Unlocked: Mastering Productivity with the PATH-C Framework

Why This Book?
As a DevSecOps Security Architect, Agile Practitioner, and DevOps Leader, I’ve spent over a decade navigating the challenges of building scalable, efficient, and secure DevOps environments. Through years of experience, research, and collaboration with teams worldwide, I developed the PATH-C Framework — a structured yet flexible approach to mastering DevOps productivity.

✅ PATH-C = Prioritization + Automation + Time-Blocking + Hyper-Focus + Continuous Learning

Each pillar is designed to tackle a specific daily challenge that DevOps professionals face. Whether you’re a beginner or a seasoned DevOps engineer, this book provides actionable strategies, real-world examples, and step-by-step guidance to help you unlock your full potential.

What You’ll Learn in This Book
📌 Prioritization: Learn how to separate urgent from important tasks, reduce context switching, and manage DevOps backlogs effectively using the Eisenhower Matrix and agile prioritization frameworks.

📌 Automation: Discover how to automate repetitive tasks, optimize CI/CD pipelines, and leverage Infrastructure as Code (IaC) to eliminate manual errors and scale effortlessly.

📌 Time-Blocking: Master structured work sessions to avoid burnout, improve focus, and align daily work with long-term objectives — using proven scheduling techniques.

📌 Hyper-Focus: Explore deep work strategies, including the Pomodoro Technique and digital tools that enhance concentration while minimizing distractions.

📌 Continuous Learning: Stay ahead in the fast-evolving DevOps field by adopting a learning mindset, leveraging online courses, certifications, and real-world problem-solving.

📌 Leadership & Collaboration: Learn how to implement PATH-C at a team level, foster accountability, and build a high-performance DevOps culture.

📌 Gamification & Motivation in DevOps: Transform DevOps tasks into engaging challenges using leaderboards, reward systems, and skill-based learning paths.

📌 Future of DevOps: Get ahead of industry trends, including AI-driven automation, GitOps, platform engineering, and cloud-native security.

Who Should Read This?
🔹 DevOps Engineers & SREs — Seeking productivity hacks to optimize workflows.
🔹 Software Developers — Want to integrate DevOps best practices into their development process.
🔹 IT Managers & Leaders — Looking for scalable frameworks to improve team performance.
🔹 DevSecOps & Cloud Architects — Interested in securing and automating software delivery.
🔹 Tech Enthusiasts & Career Changers — Exploring DevOps as a career path and looking for practical strategies to fast-track their success.

Why DevOps Professionals Love This Book
📖 “This book provides a structured approach to DevOps productivity that is both practical and insightful. The PATH-C Framework is a game-changer for DevOps teams looking to optimize their workflows and reduce inefficiencies.” — ⭐⭐⭐⭐⭐

📖 “Finally, a book that doesn’t just talk about DevOps tools, but teaches how to manage time, focus on critical tasks, and build a sustainable career in DevOps.” — ⭐⭐⭐⭐⭐

📖 “As someone managing a DevOps team, this book helped me redefine how we structure our work. PATH-C is now a core part of our team’s daily execution strategy.” — ⭐⭐⭐⭐⭐

🔹 Ready to Unlock DevOps Productivity?
📌 Whether you’re an experienced DevOps engineer or just starting, this book will give you the strategies, tools, and mindset needed to thrive in DevOps.

📌 Stop wasting time on inefficiencies, and start working smarter with PATH-C!

📖 📢 Get your copy today!
👉 Paperback & eBook available now on Amazon: DevOps Unlocked: Mastering Productivity with the PATH-C Framework | https://a.co/d/jk2dfk1

Join the DevOps Productivity Revolution!
💬 Have questions or insights? Drop a comment below!
🔁 Share this post with your network and help fellow DevOps professionals level up!
🚀 Follow me for more DevOps insights, tips, and career advice!

DevOps #Productivity #Automation #DevSecOps #CloudComputing #Agile #SRE #Kubernetes #CI/CD #DevOpsLeadership

Ethical Considerations for Generative AI in DevOps: Building Trust and Ensuring Fairness

DevOps4Me Global — Sat, 17 Feb 2024 15:26:58 +0000

Introduction

Hey, fellow DevOps enthusiasts! On February 15, 2024, I've been selected for the MBOT ( Board of Technologists (MBOT), ThursWeb online talk event, I've devled deep into a topic that's close to my heart and crucial for our field: the ethical integration of generative AI in . It Revolutionizing with a Focus on Ethical Challenges and Ensuring Trust and Fairness.As we embark on this journey, we aim to understand and integrate trust and fairness in industry-changing technology. Imagine a world where AI not only automates but also enhances every aspect of our practices, from deployment to infrastructure management, with a keen eye on ethics. Sounds dreamy, right?

*The 4 AI Waves by *

In his book AI, he discusses the four waves of AI. These waves illustrate the evolution of AI applications, reflecting an increase in complexity and integration into daily operations, directly relevant to ethical considerations as AI becomes more autonomous and ingrained in.

The Ethical Compass

To navigate these waters, we need to anchor ourselves to a robust ethical framework. This means actively working to eliminate training data and model outputs, ensuring transparency in AI , and upholding privacy and security standards that protect sensitive data.

Mitigation Biases in DevOps

The path towards responsible AI in DevOps is paved with transparency and fairness. It's about making AI's decision-making process as understandable as the code we write, ensuring that every stakeholder can trust the AI systems we deploy.

Bridging the Gap with Responsible AI

Building an ethical AI culture in DevOps isn't just about protocols and guidelines; it's about fostering a community that values awareness, supports ethical practices, and embeds these principles into the very fabric of our workflows.

Security and Privacy at Stake

In DevOps, data breaches are related to leaks in a continuous delivery pipeline. We prioritize protecting user data integrity at every stage of the CI/CD process.

Creating Clear Rules for AI

Just like there are rules for playing games fairly, we make clear rules for AI that respect both excellent values and technical needs

Top Ethical Bodies for AI

As we look ahead, the goal is clear: to advance technology while upholding our shared ethical values. It's about creating a collaborative effort towards responsible AI that benefits society at large, ensuring that the advancements we make in DevOps through AI are transparent, fair, and beneficial for all.

Conclusion

Deploy AWS Resources in Different AWS Account and Multi-Region with Terraform Multi-Provider and Alias

DevOps4Me Global — Wed, 05 Apr 2023 02:59:44 +0000

Introduction

As a company with a multi-account AWS setup, we recently faced the challenge of applying our terraform scripts across two AWS accounts, with some resources being created in one account and others in another. Fortunately, we discovered that Terraform provides a simple solution for this problem through the use of provider aliases.

By creating aliases, we were able to have multiple AWS providers within the same terraform module. This functionality can be used in a variety of situations, such as creating resources in different regions of the same AWS account or in different regions of different AWS accounts.

In this article, we will explain how to use provider aliases to create resources in single and multiple AWS accounts. By doing so, we hope to help others who may be facing similar challenges in their own multi-account AWS setups.

What is Terraform Provider Alias

In some cases, it may be necessary to define multiple configurations for the same provider and choose which one to use on a per-resource or per-module basis. This is often required when working with cloud platforms that have multiple regions, but can also be used in other situations, such as targeting multiple Docker or Consul hosts.

To create multiple configurations for a provider, simply include multiple provider blocks with the same provider name. For each additional configuration, use the "alias" meta-argument to provide a unique name segment. By doing so, you can easily select the appropriate configuration for each resource or module, making it easier to manage complex infrastructure setups. For Example;

# The default provider configuration; resources that begin with `aws_` will use
# it as the default, and it can be referenced as `aws`.
provider "aws" {
  region = "ap-southeast-1"
}

# Additional provider configuration for west coast region; resources can
# reference this as `aws.uswest1`.
provider "aws" {
  alias  = "uswest1"
  region = "us-east-1"
}

Terraform Provider Alias Use-Cases

Multiple AWS Account
Multiple Region with same AWS Account

Prequisites

As for this article, I have 2 AWS account and I configured both AWS account profile like below:

do4m-main

do4m-dev

We will used "Profile" as we going to do the 1st use-case above.

Configuring Multiple AWS providers

If you need to create resources in multiple AWS accounts using Terraform, you may run into an issue where you can't write two or more providers with the same name, such as two AWS providers. However, Terraform provides a solution to this problem by allowing you to use the "alias" argument.

With the alias argument, you can set up multiple AWS providers and use them for creating resources in both AWS accounts. This approach enables you to differentiate between the providers and avoid conflicts that could arise from having two providers with the same name. By using this technique, you can effectively manage your infrastructure and ensure that your resources are deployed to the correct AWS accounts.

I've set my provider.tf file like below;

I used do4m-main profile and set my alias as awsmain which I want to deploy to
I used do4m-dev profile and set my alias as awsdev

Defining two providers for two different AWS accounts is a great start, but it's important to know how Terraform will know which resource to create in which AWS account. To achieve this, we need to know how to refer to these different providers when defining our resources in Terraform templates.

To differentiate between providers, we need to use the "alias" argument that we defined earlier. By specifying the alias name in our resource block, we can instruct Terraform to create that resource using the provider with the corresponding alias.

For example, if we have two providers named "aws" with aliases "awsmain" and "awsdev", respectively, we can create an EC2 instances in the Main and DEV account by using the following resource block:

resource "aws_instance" "ec2_main" {
  provider        = aws.awsmain
  ami             = var.ami_apsoutheast
  instance_type   = "t2.micro"
  key_name        = "linux-sea-key"
  security_groups = ["ansible-sg"]
  tags = {
    Name    = "account-main",
    Project = "multiprovider",
    Region  = "ap-southeast-1"
  }
}

resource "aws_instance" "ec2_dev" {
  provider        = aws.awsdev
  ami             = var.ami_useast
  instance_type   = "t2.micro"
  key_name        = "linux-useast-key"
  security_groups = ["ansible-sg"]
  tags = {
    Name    = "account-dev",
    Project = "multiprovider",
    Region  = "ap-southeast-1"
  }
}

I used a Linux AMI for this example and since we defined our AMI ID with variables, so we create another file called variables.tf and fill all the AMI ID respectively in Singapore and US East region:

variable "ami_apsoutheast" {
  type    = string
  default = "ami-0af2f764c580cc1f9"
}

variable "ami_useast" {
  type    = string
  default = "ami-00c39f71452c08778"
}

Let's get started!: Terraform Apply

Now we already set up all providers we want, we can run all the Terraform commands below:

terraform init

terraform plan

➜  terraform-multi-providers terraform plan   

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # aws_instance.ec2_dev will be created
  + resource "aws_instance" "ec2_dev" {
      + ami                                  = "ami-0af2f764c580cc1f9"
      + arn                                  = (known after apply)
      + associate_public_ip_address          = (known after apply)
      + availability_zone                    = (known after apply)
      + cpu_core_count                       = (known after apply)
      + cpu_threads_per_core                 = (known after apply)
      + disable_api_stop                     = (known after apply)
      + disable_api_termination              = (known after apply)
      + ebs_optimized                        = (known after apply)
      + get_password_data                    = false
      + host_id                              = (known after apply)
      + host_resource_group_arn              = (known after apply)
      + iam_instance_profile                 = (known after apply)
      + id                                   = (known after apply)
      + instance_initiated_shutdown_behavior = (known after apply)
      + instance_state                       = (known after apply)
      + instance_type                        = "t2.micro"
      + ipv6_address_count                   = (known after apply)
      + ipv6_addresses                       = (known after apply)
      + key_name                             = "linux-sea-key"
      + monitoring                           = (known after apply)
      + outpost_arn                          = (known after apply)
      + password_data                        = (known after apply)
      + placement_group                      = (known after apply)
      + placement_partition_number           = (known after apply)
      + primary_network_interface_id         = (known after apply)
      + private_dns                          = (known after apply)
      + private_ip                           = (known after apply)
      + public_dns                           = (known after apply)
      + public_ip                            = (known after apply)
      + secondary_private_ips                = (known after apply)
      + security_groups                      = [
          + "ansible-sg",
        ]
      + source_dest_check                    = true
      + subnet_id                            = (known after apply)
      + tags                                 = {
          + "Name"    = "account-dev"
          + "Project" = "multiprovider"
          + "Region"  = "ap-southeast-1"
        }
      + tags_all                             = {
          + "Name"    = "account-dev"
          + "Project" = "multiprovider"
          + "Region"  = "ap-southeast-1"
        }
      + tenancy                              = (known after apply)
      + user_data                            = (known after apply)
      + user_data_base64                     = (known after apply)
      + user_data_replace_on_change          = false
      + vpc_security_group_ids               = (known after apply)

      + capacity_reservation_specification {
          + capacity_reservation_preference = (known after apply)

          + capacity_reservation_target {
              + capacity_reservation_id                 = (known after apply)
              + capacity_reservation_resource_group_arn = (known after apply)
            }
        }

      + ebs_block_device {
          + delete_on_termination = (known after apply)
          + device_name           = (known after apply)
          + encrypted             = (known after apply)
          + iops                  = (known after apply)
          + kms_key_id            = (known after apply)
          + snapshot_id           = (known after apply)
          + tags                  = (known after apply)
          + throughput            = (known after apply)
          + volume_id             = (known after apply)
          + volume_size           = (known after apply)
          + volume_type           = (known after apply)
        }

      + enclave_options {
          + enabled = (known after apply)
        }

      + ephemeral_block_device {
          + device_name  = (known after apply)
          + no_device    = (known after apply)
          + virtual_name = (known after apply)
        }

      + maintenance_options {
          + auto_recovery = (known after apply)
        }

      + metadata_options {
          + http_endpoint               = (known after apply)
          + http_put_response_hop_limit = (known after apply)
          + http_tokens                 = (known after apply)
          + instance_metadata_tags      = (known after apply)
        }

      + network_interface {
          + delete_on_termination = (known after apply)
          + device_index          = (known after apply)
          + network_card_index    = (known after apply)
          + network_interface_id  = (known after apply)
        }

      + private_dns_name_options {
          + enable_resource_name_dns_a_record    = (known after apply)
          + enable_resource_name_dns_aaaa_record = (known after apply)
          + hostname_type                        = (known after apply)
        }

      + root_block_device {
          + delete_on_termination = (known after apply)
          + device_name           = (known after apply)
          + encrypted             = (known after apply)
          + iops                  = (known after apply)
          + kms_key_id            = (known after apply)
          + tags                  = (known after apply)
          + throughput            = (known after apply)
          + volume_id             = (known after apply)
          + volume_size           = (known after apply)
          + volume_type           = (known after apply)
        }
    }

  # aws_instance.ec2_main will be created
  + resource "aws_instance" "ec2_main" {
      + ami                                  = "ami-0af2f764c580cc1f9"
      + arn                                  = (known after apply)
      + associate_public_ip_address          = (known after apply)
      + availability_zone                    = (known after apply)
      + cpu_core_count                       = (known after apply)
      + cpu_threads_per_core                 = (known after apply)
      + disable_api_stop                     = (known after apply)
      + disable_api_termination              = (known after apply)
      + ebs_optimized                        = (known after apply)
      + get_password_data                    = false
      + host_id                              = (known after apply)
      + host_resource_group_arn              = (known after apply)
      + iam_instance_profile                 = (known after apply)
      + id                                   = (known after apply)
      + instance_initiated_shutdown_behavior = (known after apply)
      + instance_state                       = (known after apply)
      + instance_type                        = "t2.micro"
      + ipv6_address_count                   = (known after apply)
      + ipv6_addresses                       = (known after apply)
      + key_name                             = "linux-sea-key"
      + monitoring                           = (known after apply)
      + outpost_arn                          = (known after apply)
      + password_data                        = (known after apply)
      + placement_group                      = (known after apply)
      + placement_partition_number           = (known after apply)
      + primary_network_interface_id         = (known after apply)
      + private_dns                          = (known after apply)
      + private_ip                           = (known after apply)
      + public_dns                           = (known after apply)
      + public_ip                            = (known after apply)
      + secondary_private_ips                = (known after apply)
      + security_groups                      = [
          + "ansible-sg",
        ]
      + source_dest_check                    = true
      + subnet_id                            = (known after apply)
      + tags                                 = {
          + "Name"    = "account-main"
          + "Project" = "multiprovider"
          + "Region"  = "ap-southeast-1"
        }
      + tags_all                             = {
          + "Name"    = "account-main"
          + "Project" = "multiprovider"
          + "Region"  = "ap-southeast-1"
        }
      + tenancy                              = (known after apply)
      + user_data                            = (known after apply)
      + user_data_base64                     = (known after apply)
      + user_data_replace_on_change          = false
      + vpc_security_group_ids               = (known after apply)

      + capacity_reservation_specification {
          + capacity_reservation_preference = (known after apply)

          + capacity_reservation_target {
              + capacity_reservation_id                 = (known after apply)
              + capacity_reservation_resource_group_arn = (known after apply)
            }
        }

      + ebs_block_device {
          + delete_on_termination = (known after apply)
          + device_name           = (known after apply)
          + encrypted             = (known after apply)
          + iops                  = (known after apply)
          + kms_key_id            = (known after apply)
          + snapshot_id           = (known after apply)
          + tags                  = (known after apply)
          + throughput            = (known after apply)
          + volume_id             = (known after apply)
          + volume_size           = (known after apply)
          + volume_type           = (known after apply)
        }

      + enclave_options {
          + enabled = (known after apply)
        }

      + ephemeral_block_device {
          + device_name  = (known after apply)
          + no_device    = (known after apply)
          + virtual_name = (known after apply)
        }

      + maintenance_options {
          + auto_recovery = (known after apply)
        }

      + metadata_options {
          + http_endpoint               = (known after apply)
          + http_put_response_hop_limit = (known after apply)
          + http_tokens                 = (known after apply)
          + instance_metadata_tags      = (known after apply)
        }

      + network_interface {
          + delete_on_termination = (known after apply)
          + device_index          = (known after apply)
          + network_card_index    = (known after apply)
          + network_interface_id  = (known after apply)
        }

      + private_dns_name_options {
          + enable_resource_name_dns_a_record    = (known after apply)
          + enable_resource_name_dns_aaaa_record = (known after apply)
          + hostname_type                        = (known after apply)
        }

      + root_block_device {
          + delete_on_termination = (known after apply)
          + device_name           = (known after apply)
          + encrypted             = (known after apply)
          + iops                  = (known after apply)
          + kms_key_id            = (known after apply)
          + tags                  = (known after apply)
          + throughput            = (known after apply)
          + volume_id             = (known after apply)
          + volume_size           = (known after apply)
          + volume_type           = (known after apply)
        }
    }

Plan: 2 to add, 0 to change, 0 to destroy.

terraform apply

As the result/outcome from terraform apply, we have both EC2 instance deployed in both AWS account we set earlier:

AWS-MAIN(Singapore):

AWS-DEV(US East-Virginia)

Deploy EC2 Instance in multi-region with same AWS Account

Now we proceed with our 2ns use-case, we want to use Terraform Provider Alias to deploy 2 EC2 intances into different AWS Region with my MAIN AWS account. For this use-case I have the following scenario:

I want to deploy the 1st EC2 to Singapore region (ap-southeast-1)
I want to deploy the 2nd EC2 to Sydney region (ap-southeast-2)

Create new folder called multi-region in current code workspace:

mkdir multi-region
cd multi-region

Next, we create our provider.tf with alias like below:

provider "aws" {
  alias  = "se1"
  region = "ap-southeast-1"
}

provider "aws" {
  alias  = "se2"
  region = "ap-southeast-2"
}

Now we have 2 terraform providers:

I've created 1st terraform alias se1 and region is Singapore (ap-southeast-1)
Another terraform alias is se2 and region is Sydney ap-southeast-2

We can re-use/clone the earlier main.tf file we created in 1st use-case and since already know AMI ID I want to used;you can modified like below:

resource "aws_instance" "ec2_main" {
  provider        = aws.se1
  ami             = "ami-0af2f764c580cc1f9"
  instance_type   = "t2.micro"
  key_name        = "linux-sea-key"
  security_groups = ["ansible-sg"]
  tags = {
    Name    = "account-se1",
    Project = "multiprovider",
    Region  = "ap-southeast-1"
  }
}

resource "aws_instance" "ec2_dev" {
  provider        = aws.se2
  ami             = "ami-0d0175e9dbb94e0d2"
  instance_type   = "t2.micro"
  key_name        = "linux-sea-key"
  security_groups = ["ansible-sg"]
  tags = {
    Name    = "account-se2",
    Project = "multiprovider",
    Region  = "ap-southeast-2"
  }
}

Lastly, we can run the Terraform commands below to see the output of above setup:

terraform init

terraform plan

terraform plan

After all terraform commands executed, we get both EC2 instances in both regions below:

Singapore Region

Sydney Region

Conclusion

As we strive to scale our application and infrastructure, it's important to adopt an effective approach when working with Terraform. Relying on a single state file can lead to potential bottlenecks and even a point of failure, especially when managing a large infrastructure team. However, we can overcome these challenges by using multiple AWS providers in different combinations.

This approach enables us to manage numerous Terraform state files and carry out multi-region deployment. The same principles can also apply when using different types of providers in the same Terraform module, such as AWS and GCP. By embracing this strategy, we can streamline our infrastructure management processes and optimize our workflow. This ensures that our application and infrastructure are fully scalable, flexible, and resilient, regardless of the size of our infrastructure team or the types of providers we use.

Create an API with a private integration to an AWS ECS service with Terraform (IaC)

DevOps4Me Global — Mon, 13 Feb 2023 15:30:28 +0000

Introduction

You may connect Amazon API Gateway API routes to VPC-restricted resources using VPC links. A VPC connection is an abstraction layer on top of other networking resources and functions like any other integration endpoint for an application programming interface (API). This makes it easier to set up secure connections.

Use-case

We want to setup a private AWS API Gateway to our backend service that used AWS serverless service such a Fargate and it's deployed in AWS ECS Cluster. The architecture below that we want to create in this blog post.

Steps

Terraform
We need to create our main Terraform file(main.tf); proceed to execute below command in your prefer directory.

mkdir automation && cd automation
vi main.tf

On the top we set our AWS Availability Zone (AZ):

data "aws_availability_zones" "available_zones" {
  state = "available"
}

We also need to create our versions.tf file as below:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 4.0"
    }
  }
}
provider "aws" {
  region = "ap-southeast-1"
  default_tags {
    tags = {
      Name = "do4m-demo"
    }
  }
}

Then, we need output.tf file to get the
Lastly, for our prerequisites step, we create variables.tf file for all required Fargate application count we neeed.

variable "app_count" {
  type    = number
  default = 1
}

VPC Setup
First, you use a Terraform create a Amazon VPC which includig VPC subnets(Private & Public), Internet Gateway, NAT Gateway for Private subnets and route table for subnets association.

#VPC Setting
resource "aws_vpc" "default" {
  cidr_block = "10.32.0.0/16"
}

resource "aws_subnet" "public" {
  count                   = 2
  cidr_block              = cidrsubnet(aws_vpc.default.cidr_block, 8, 2 + count.index)
  availability_zone       = data.aws_availability_zones.available_zones.names[count.index]
  vpc_id                  = aws_vpc.default.id
  map_public_ip_on_launch = true
}

resource "aws_subnet" "private" {
  count             = 2
  cidr_block        = cidrsubnet(aws_vpc.default.cidr_block, 8, count.index)
  availability_zone = data.aws_availability_zones.available_zones.names[count.index]
  vpc_id            = aws_vpc.default.id
}

resource "aws_internet_gateway" "gateway" {
  vpc_id = aws_vpc.default.id
}

resource "aws_route" "internet_access" {
  route_table_id         = aws_vpc.default.main_route_table_id
  destination_cidr_block = "0.0.0.0/0"
  gateway_id             = aws_internet_gateway.gateway.id
}

resource "aws_eip" "gateway" {
  count      = 2
  vpc        = true
  depends_on = [aws_internet_gateway.gateway]
}

resource "aws_nat_gateway" "gateway" {
  count         = 2
  subnet_id     = element(aws_subnet.public.*.id, count.index)
  allocation_id = element(aws_eip.gateway.*.id, count.index)
}

resource "aws_route_table" "private" {
  count  = 2
  vpc_id = aws_vpc.default.id

  route {
    cidr_block     = "0.0.0.0/0"
    nat_gateway_id = element(aws_nat_gateway.gateway.*.id, count.index)
  }
}

resource "aws_route_table_association" "private" {
  count          = 2
  subnet_id      = element(aws_subnet.private.*.id, count.index)
  route_table_id = element(aws_route_table.private.*.id, count.index)
}

Security Group

resource "aws_security_group" "lb" {
  name   = "do4m-alb-sg"
  vpc_id = aws_vpc.default.id

  ingress {
    protocol    = "tcp"
    from_port   = 80
    to_port     = 80
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

AWS ALB Setting

resource "aws_lb" "default" {
  name            = "do4m-lb"
  subnets         = aws_subnet.public.*.id
  security_groups = [aws_security_group.lb.id]
}

resource "aws_lb_target_group" "hello_world" {
  name        = "do4m-target-group"
  port        = 80
  protocol    = "HTTP"
  vpc_id      = aws_vpc.default.id
  target_type = "ip"
}

resource "aws_lb_listener" "hello_world" {
  load_balancer_arn = aws_lb.default.id
  port              = "80"
  protocol          = "HTTP"

  default_action {
    target_group_arn = aws_lb_target_group.hello_world.id
    type             = "forward"
  }
}

AWS ECS and Fargate Setting

resource "aws_ecs_task_definition" "hello_world" {
  family                   = "hello-world-app"
  network_mode             = "awsvpc"
  requires_compatibilities = ["FARGATE"]
  cpu                      = 1024
  memory                   = 2048

  container_definitions = <<DEFINITION
[
  {
    "image": "registry.gitlab.com/architect-io/artifacts/nodejs-hello-world:latest",
    "cpu": 1024,
    "memory": 2048,
    "name": "hello-world-app",
    "networkMode": "awsvpc",
    "portMappings": [
      {
        "containerPort": 3000,
        "hostPort": 3000
      }
    ]
  }
]
DEFINITION
}

resource "aws_security_group" "hello_world_task" {
  name   = "do4m-task-sg"
  vpc_id = aws_vpc.default.id

  ingress {
    protocol        = "tcp"
    from_port       = 3000
    to_port         = 3000
    security_groups = [aws_security_group.lb.id]
  }

  egress {
    protocol    = "-1"
    from_port   = 0
    to_port     = 0
    cidr_blocks = ["0.0.0.0/0"]
  }
}

resource "aws_ecs_cluster" "main" {
  name = "do4m-cluster"
}

resource "aws_ecs_service" "hello_world" {
  name            = "hello-world-service"
  cluster         = aws_ecs_cluster.main.id
  task_definition = aws_ecs_task_definition.hello_world.arn
  desired_count   = var.app_count
  launch_type     = "FARGATE"

  network_configuration {
    security_groups = [aws_security_group.hello_world_task.id]
    subnets         = aws_subnet.private.*.id
  }

  load_balancer {
    target_group_arn = aws_lb_target_group.hello_world.id
    container_name   = "hello-world-app"
    container_port   = 3000
  }

  depends_on = [aws_lb_listener.hello_world]
}

AWS API Gateway and VPC PrivateLink Setting

#1: API Gateway
resource "aws_apigatewayv2_api" "api" {
  name          = "do4m-api-gateway"
  protocol_type = "HTTP"
}
#2: VPC Link
resource "aws_apigatewayv2_vpc_link" "vpc_link" {
  name               = "development-vpclink"
  security_group_ids = [aws_security_group.lb.id]
  subnet_ids         = aws_subnet.private.*.id
}
#3: API Integration
resource "aws_apigatewayv2_integration" "api_integration" {
  api_id             = aws_apigatewayv2_api.api.id
  integration_type   = "HTTP_PROXY"
  connection_id      = aws_apigatewayv2_vpc_link.vpc_link.id
  connection_type    = "VPC_LINK"
  description        = "VPC integration"
  integration_method = "ANY"
  integration_uri    = aws_lb_listener.hello_world.arn
  depends_on         = [aws_lb.default]
}
#4: APIGW Route
resource "aws_apigatewayv2_route" "default_route" {
  api_id    = aws_apigatewayv2_api.api.id
  route_key = "$default"
  target    = "integrations/${aws_apigatewayv2_integration.api_integration.id}"
}
#5: APIGW Stage
resource "aws_apigatewayv2_stage" "default_stage" {
  api_id      = aws_apigatewayv2_api.api.id
  name        = "$default"
  auto_deploy = true
}

Execute

First, we need to configure AWS account before we can run Terraform:

aws configure

Then we run command to initial our Terraform modules:

terraform init

After that, we run validation and formating command below:

terraform validate && terraform fmt

Next, we run command to create Terraform Plan:

terraform plan -no-color > tfplan.txt

Lastly, once we confirmed and validated our plan, we can execute Terraform Apply command to create all the AWS resources/services we set above:

terraform apply -auto-approve

Output

VPC
Subnets

Route Table

Elastic Public IP

Internet Gateway

NAT Gateway

ECS Cluster

Fargate Task Definition

AWS API Gateway

VPC PrivateLink

API Testing

Once your API has been created, you must test it to ensure it is functioning properly. Invoking your API from a web browser will save time and effort. In order to put your API through its paces
To use the API Gateway, log in to the console via https://console.aws.amazon.com/apigateway . Choose your API and you need to invoke URL.

The result we able to call our private services below:

Clean Up

To clean/remove all AWS resources we created in this post, we run Terraform destroy below:

terraform destroy -auto-approve

Source Code

You can refer for full source via this link: https://github.com/devops4mecode/ecs-fargate-vpclink-apigw

AWS Cost Optimization : Automatic Detect Your Unused EBS and Delete It

DevOps4Me Global — Wed, 08 Feb 2023 03:31:29 +0000

What is cost optimization in AWS?

Its primary objective is to achieve the lowest possible cost for the system or workload while operating in the AWS environment. You should try to minimise expenditures while taking into consideration the needs of your account. However, you shouldn't do this at the expense of performance, security, or dependability.

It is crucial to fully grasp the value of AWS, as well as measure and efficiently manage your AWS consumption and expenses, as you transfer workloads to AWS and grow your use of different AWS services. This is especially important when you increase your use of AWS services.

Cost Optimization Use-Case

A lack of awareness of the EBS volume lifetime results in additional expenditures for underused and neglected resources. Unexpected costs on an AWS account may result from Elastic Block Storage (EBS) volumes that aren't associated with an EC2 instance or used. Some EBS volumes may continue to exist after an EC2 instance is shut down. You are paying for EBS volumes in AWS accounts even though they are not associated. Following the below steps will allow you to save cloud charges and avoid wasted resources by removing an EBS volume that was accidentally left unattached.

Steps

Prerequisite

EBS Volume
We purposely create another 2 new AWS EBS that not attach to any EC2 instance. We use Boto3 ; which Boto3 is the Amazon Web Services (AWS) Software Development Kit (SDK) for Python which is maintained and published by Amazon Web Services. Boto3 allows Python developers to write software that makes use of services like Amazon S3 and Amazon EC2 as well as provision AWS Services from Boto3. You need to configure your local environment with your AWS Access Key ID and Access Secret Key by execute below command:

aws configure

Once you have configured your AWS account and installed the Boto3 SDK, then you may execute the Pyhton code below to create a new EBS volume for our use-case.

### Author:Najib Radzuan
### CreatedDate:23 Dec 2022
### Purpose: Create New EBS volumes
### Requirements: Boto3(EC2)

import boto3
ec2 = boto3.client('ec2')
response = ec2.create_volume(
    AvailabilityZone='ap-southeast-1a',
    Size=20,
    VolumeType='gp2',
    TagSpecifications=[
        {
            'ResourceType': 'volume',
            'Tags': [
                {
                    'Key': 'Name',
                    'Value': 'Unused_Vol1'
                },
            ]
        },
    ],
)

We had created 2 new EBS volumes and it not attach to any EC2 instance as shown below.

AWS IAM Role

Step 1
We need to create a new AWS Simple Notification Service (SNS) for our use-case notification whenever the AWS Lambda function detected unused EBS and deleted it. Enter the following input for our AWS SNS Topic:

Type: Standard
Name: Notify-Unused-EBS-Volume
Display: Notify-Unused-EBS-Volume

Let the rest of the configuration to be default values.

Next, we to set/create our subscription of the newly created SNS Topic.

Topic ARN: [Newly created AWS SNS Topic ARN]
Protocol: Email
Endpoint: [your subscriber email]

Let the rest to be default values.

Then, we need confirmation from the subscriber we set above via email below:

Once the subscriber clicked the "Confirmation subscription", they we redierected below page that said they are subscribed to the AWS SNS Topic:

Finally, we have new subscriber to our AWS SNS Topic:

Don't forget to copy and paste the AWS SNS Topic ARN somewhere, since we going to use it in AWS Lambda Function code later for SNS notification.

Step 2

Create a new AWS Lamda Function and you can enter below configuration for our new AWS Lambda Funtion:

Author from scratch
Function name: do4m-unused-volume
Runtime : Pyhton 3.9
Architecture: x86_64

Left the rest configuration to be default values.

Copy and paste below code into your Lambda Function code. There is 2 parts in this code:

1) Boto3

We use Boto3 EC2 module to find all unsed EBS volumnes in the region we set in the code.
We put the detected unused EBS ID in "Attachement" for SNS notification usage.
Once we get all the unused EBS IDs, then we delete EBS volume that had mark with “Available” status.

2) AWS SNS

We get from "Attachment" unused EBS IDs and we send out the SNS email notification to our subscriber(s) that we set in the previous step.

### Author:Najib Radzuan
### CreatedDate:23 Dec 2022
### Purpose: Detect all unused EBS volumes under selected region
### Requirements: Boto3(EC2),SNS Arn

import boto3
ec2 = boto3.client('ec2')
sns_client = boto3.client('sns')
volumes = ec2.describe_volumes()
ec2 = boto3.resource('ec2',region_name='ap-southeast-1')

def lambda_handler(event, context):
    #Get All Unused Volume
    unused_volumes = []
    for vol in volumes['Volumes']:
        if len(vol['Attachments']) == 0:
            vol1 = ("-----Unused Volume ID = {}------".format(vol['VolumeId']))
            unused_volumes.append(vol1)

    #Delete Volume if Unused
    for vol in ec2.volumes.all():
        if  vol.state=='available':
                vid=vol.id
                v=ec2.Volume(vol.id)
                v.delete()
                print ('Deleted ' +vid)
        ## If we use tagging as ##
        # continue
        # for tag in vol.tags:
        #     if tag['Key'] == 'Name':
        #         value = tag['Value']
        #         if value != 'DND' and vol.state == 'available':
        #             vid = vol.id
        #             v = ec2.Volume(vol.id)
        #             v.delete()
        #             print('Deleted ' + vid)

    #email
    sns_client.publish(
        TopicArn='arn:aws:sns:ap-southeast-1:627315336549:Notify-Unused-EBS-Volume',
        Subject='Warning - Unused Volume List',
        Message=str(unused_volumes)
    )
    return "success"

Step 3
The final step we going to set our AWS EventBridge Rule for schedule to trigger our AWS Lambda Function we created in the previous step. Go to AWS EventBridge->Create New Rule; we enter below input.

Name: do4m-unused-ebs-rule
Description: do4m-unused-ebs-rule
Event Bus: default
Rule Type: Schedule Proceed with "Continue to create rule"

We set our EventBridge Schedule Pattern. We set our Cron Expression as below. For this use-case; I created schedule patter for every Saturday at 8 AM it will triggered Lambda and it happen every month every year. You may change to your requirement and you can refer the cron-expression here.

Next, we choose our Target for EventBridge Rule as invoke the AWS Lambda. We select the AWS Lambda Function we created in the previous step as our Lambda Function below.

Lastly, we created our AWS EventBridge Rule for Scheduler.

Note
For sample I've changed to trigger our AWS Lambda function every 5 minutes to get fast result.

The Results

We can monitor whether our EventBridge Rule is work by go to CloudWatch Log group we created via AWS Lambda Function. As we can see CloudWatch log record below it detected the unused EBS volumes and deleted it.

Our EBS Volume is deleted and only left the "In-Use" EBS volume in AWS Console.

We also get SNS notification that our AWS Lambda Function detected the unused and deleted it.

Summary

In this post we learn how to do AWS Cost Optimization by auto detect your unused EBS volume via AWS Lambda Function which it triggered by AWS EventBridge Rule. Lastly we get AWS SNS notifition email whenever Lambda Function found unused EBS volume.

Deploy Apps using GitHub Actions to AWS Beanstalk

DevOps4Me Global — Wed, 17 Aug 2022 05:46:00 +0000

Introduction

In this post I will show you how you use the GitHub Actions and Amazon Web Services (AWS)-Beanstalk, we'll build a CI/CD pipeline from the ground up. I've broken down the guide into three sections for easier reading and comprehension:

Before getting too bogged down in jargon, let's define a few key terms.

Second, we'll implement continuous integration (CI) to ensure that all builds and tests are executed without human intervention.

Our code will be automatically deployed to AWS after we have set up continuous delivery (CD).

Yes, that was a lot to take in. First, let's define each of these terms in detail so you know exactly where we're headed.

Definition
What is GitHub Action?
I'm going to oversimplify the GitHub Actions concept because I can't think of a better way to explain it. Hence, I've made below Mind Map for better understanding on GitHub Action.

For more details on GitHub , please go to GitHub Action Details

Goals

How to automatically build and run unit tests on push or on PR to the main branch with GitHub Actions.
How to automatically deploy to AWS on push or on PR to the main branch with GitHub Actions.

PART 1: How to Automatically Run Builds and Tests - Continuous Integration (CI)

Prerequisites
P1:P1 Project Folder & Files Structure
I have created a demo Django project which you can grab from this repository:

git clone https://github.com/devops4mecode/django-github-actions-aws.git

After you download the code, create a virtualenv and install the requirements via pip:

pip install -r requirements.txt

Part1:P2 Locally Build and Unit-Test
We build and test the Django App locally below;

Let's set up GitHub Actions now that you have a local Django project ready to go.

Part1:P3 Create GitHub Repository for Django App

We need tp create a new GitHub repository, once create you will get below repository. You may use different name than mine.

As you can see, we also don't have GitHub Action configure yet.

I also created a new DEV branch and update all code/script that we need for GitHub Actions.

STEPS:
Part1:S1 How to Configure GitHub Actions

Our project is now ready to go. More importantly, however, we have committed our polished update to GitHub and have a testcase prepared for the view that we have defined.

Each time a pull request or push is made to master, we want GitHub to automatically initiate a build and run all of our tests. The build and testing weren't triggered by GitHub Actions after we published to main.

Make a new folder in your.github called workflows: The YAML file editor is where all your files will be made. In order to get our build and test routines up and running, we need to make a first workflow. To accomplish this, we make a new file ending in.yml. Put the name "build test.yml" in this file's name. Incorporate the following into the newly produced yaml file:

name: Build and Test

on:
  push:
    branches: [master]
  pull_request:
    branches: [master]

jobs:
  test:
    runs-on: ubuntu-latest

    steps:
    - name: Checkout code
      uses: actions/checkout@v2
    - name: Set up Python Environment
      uses: actions/setup-python@v2
      with:
        python-version: '3.x'
    - name: Install Dependencies
      run: |
        python -m pip install --upgrade pip
        pip install -r requirements.txt

    - name: Run Tests
      run: |
        python manage.py test
  deploy:
    needs: [test]
    runs-on: ubuntu-latest

    steps:
    - name: Checkout GitHub source code
      uses: actions/checkout@v2

    - name: Generate deployment package
      run: zip -r deploy.zip . -x '*.git*'

    - name: Deploy to AWS Elastic Beanstalk
      uses:  einaregilsson/beanstalk-deploy@v20
      with:
        aws_access_key: ${{ secrets.AWS_ACCESS_KEY_ID }}
        aws_secret_key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        application_name: django-github-actions-aws
        environment_name: Djangogithubactionsaws-env
        version_label: "ver-${{ github.sha }}"
        use_existing_version_if_available: true
        region: "ap-southeast-1"
        deployment_package: deploy.zip

Part1:S2 Push/PR DEV branch code to GitHub Repository

Now that you've defined a workflow by adding the config file in the designated folder, you can commit and push your change to your remote repo.

Let's create Pull Request and Merge it to Master branch.

While our PR being check, you can see our GitHub Action already running;

If you navigate to the Actions tab of your remote repo, you should see a workflow with the name Build and Test (the name which we've given it) listed there. As of now, we can see our Deploy jobs is failed, since we not setup yet our AWS Beanstalk. So this output is expected, don't worry, we will setup it later.

PART 2: How to Automatically Deploy Our Code to AWS - Continuous Deployment (CD)

Prerequisites

Part2:P1 AWS AWS_ACCESS_KEY_ID & AWS_SECRET_ACCESS_KEY
Go to AWS Management Console -> IAM -> User -> Access Key

Please copy and paste Access Key and Secret to your Notepad/any place you prefered, since we will reuse it when we want to setup our secret in GitHub repository later.

Part2:P2 Elastic Beanstalk environment
Before showing the workflow, we must have an operating environment to receive our Django app from GitHub Actions. Since we will use DJango Python as our language, hence, create a new Environment as below.
3.1 Create a new Beanstalk applicaton

Finally, we have created our Beanstalk Sample App

STEPS:

Part2:S1 Configure GitHub Secret
Previously, we have copy our AWS Access Key and Secret Key, now go to GitHub Repository Setting. Then, go to Secret below.

Choose "Action" menu, then click "New repository secret" button;

Give your secret name, please follow back what you had defined in our workflow script above.

AWS_ACCESS_KEY_ID:

AWS_SECRET_ACCESS_KEY:

Part2:S2 Configure your Project for Elastic Beanstalk

Our project's application.py file is the default location where Elastic Beanstalk will look for our code. Our application cannot run without that file, but it is not included in our repository. The Elastic Beanstalk must be instructed to use the wsgi.py file in our project rather than the default python servlet container. Here's what you need to do:

In the root of your project's directory, make a new folder and label it.ebextensions. Make a configuration file in that directory. What you want to call it is up to you. My file is called eb.config. The following should be included in your configuration file:

option_settings:
  aws:elasticbeanstalk:container:python:
    WSGIPath: django_github_actions_aws.wsgi:application

Part2:S3 Update and finalize your Workflow File

Please update all 5 parameters below with your parameters:

Your AWS Access Key.
Your AWS Secret Key.
Beanstalk's Application Name.
Beanstalk's Environment Name.
AWS Region that want to deploy.

This step utilizes einaregilsson/beanstalk-deploy@v20. Actions are reusable apps that handle often repeated activities. the e inaregilsson/beanstalk-deploy@v20 is one.m To emphasize the above, remember our deployment step:

GitHub->AmazonS3->Elastic Beanstalk

We didn't set up Amazon s3 in this tutorial. We didn't upload to or pull from an S3 bucket in our workflow file. The einaregilsson/beanstalk-deploy@v20 operation performs all that for us. You can construct your own action to handle monotonous activities and sell it on GitHub Marketplace.

Now that you've updated your workflow file locally, you can then commit and push this change to your remote. Your jobs will run and your code will be deployed to the Elastic Beanstalk instance you created.

You also can check the log on deploy jobs to our AWS Beanstalk below.

Lastly, we have updated our Sample App to new code below.

Conclusion
Good lord, this one was a marathon, right? I concluded by defining CI/CD Pipeline, Amazon Web Services (AWS), and GitHub Actions. We also learned how to set up GitHub Actions for automated code deployment to an AWS Elastic Beanstalk environment.

DevSecOps in AWS Codepipeline

DevOps4Me Global — Mon, 25 Apr 2022 17:08:53 +0000

Introduction
Traditional security cannot keep pace with DevOps’s lightning-fast software development cycles. Organisations need to inject continuous security and automated testing throughout the software development process to improve security. Secure DevOps—that is, DevSecOps—is about making security central to development and operations. Building security into every stage of the software pipeline fills long-standing gaps between IT and Security. DevSecOps approach helps spot software security issues faster and alleviates security bottlenecks. And, it preserves the rapid development pace that DevOps makes possible. This post will show you how to do implement DevSecOps CI/CD using AWS CodePipeline.

Before DevSecOps

Above is normal Angular website which the flow like below:

Developer/Programmer wil commit the new updated code to AWS CodeCommit Repository.
AWS CodePipeline **automatic triggered from the CodeCommit updated code in **SOURCE stage below:
Then it's goes to BUILD stage started the Unit-Test action group using "ng-test" command from "unit-test-buildspec.yml" file we set.
Then it goes to Build process by using "ng-build" via buildspec.yml file we set.
We have dynamic Staging EC2 server by using AWS CloudFormation which we only provision Staging server whenever we have new updated code.
Once we provisioned Staging server then we deploy to EC2 server. We also passed CodeDeploy parameter which new Staging URL for manual approval or reviewer to evaluate our staging environment/server.

Here our Staging look like:

Once the Approver approved and reviewed our Staging environment, we will delete our Staging CloudFormation Stack.
Lastly, we deploy to Production environment which we using AWS CodeDeploy

*From DevOps to DevSecOps *
Now we can introduce the DevSecOps tools to use "Shift-Left" approach which added security scanner in BUILD, STAGING and PRD stage like below.

DevSecOps Toolchain

How to optimize your DevSecOps Pipeline

Everyone should understand failures quickly

CI job output with key messages
Use colored error messages with emojis ✅
Hint and link troubleshooting docs

Always use exactly the same environment
A CI/CD pipeline can’t be reliable if a pipeline run modifies the next pipeline’s environment.
Each workflow should start from the same, clean, and isolated environment.
Document the pipeline design
In your repository use Markdown docs to describe what you do on your CI/CD pipeline.
You also can use Wiki Page or Confluence to make everyone in the team know better about CI/CD process.
Communication
Human communication, collaboration, and teamwork are examples of factors that do not rely on automation.
In the absence of these three factors, success will be extremely hard to achieve.
It's crucial to optimize communication and transparency in your CI/CD pipeline workflow if you want it to succeed.

Conclusion

Shift-Left + DevSecOps I think that everyone in the organization must make an effort to "Shift-Left" to a DevSecOps culture or methodologies and come up with a multidisciplinary security team.
Agile + DevSecOps DevSecOps must be fed by Agile software development. Security user stories must be part of each sprint.
Automation is the key Security and test automation can reduce delivery time, improve quality and security, and eliminate human error

DevSecOps Transformation Bucket List

DevOps4Me Global — Fri, 22 Apr 2022 05:25:08 +0000

Introduction
DevOps’s lightning-fast software development cycles imply traditional security measures can’t keep up. Organizations must include automated and continuous security testing in their software development processes to enhance security. It’s all about making security a part of the DevOps process — DevSecOps. That’s Filling the gap between I.T and security by incorporating security into every step of the software development lifecycle process Security bottlenecks may be alleviated by using the DevSecOps strategy. It also maintains the high development pace that DevOps allows. This post will share how the DevSecOps framework, activities, process, and challenges during your DevSecOps transformations.

What is DevSecOps?
DevSecOps is the practice of incorporating security and compliance testing into the DevOps pipeline without jeopardizing the speed and agility of continuous delivery. DevSecOps adoption is needed to make software delivery more agile, responsive, and secure. More team collaboration between IT security and the product team (including development and operations) must happen. In the past, the security team worked alone. There were always security and compliance checks done at the end. When you have a faster software delivery cycle with DevOps and more often make software, those checks at the end make it more difficult and slow down the continuous delivery process.

Why Move to DevSecOps?
The increased delivery velocity with a DevOps CI/CD pipeline, particularly with microservices-based applications, is prone to introducing vulnerabilities and making continuous security a major issue in organizations.
Security teams must learn and understand the security implications of all these new technologies to identify issues. The learning curve becomes a significant bottleneck, so automating security checks early in the continuous delivery process, i.e., practising DevSecOps.

DevSecOps Benefits

DevSecOps Metrics
DevSecOps is a proven method for improving the quality and security of software, which can be measured using these metrics:
• Meantime to production: the average time it takes from when new software features are required to when they are up and running.
• Average lead-time: the time it takes to deliver and deploy a new requirement.
• Deployment speed: how quickly DevSecOps can deploy a new application version into production.
• Deployment frequency: the frequency with which DevSecOps can deploy a new release into production.
• Production failure rate: the frequency with which software fails during production.
• Mean time to recovery (MTTR): the time it takes for applications in the production stage to recover from a failure.
Furthermore, DevSecOps practice enables:
• Fully automated risk characterization, monitoring, and mitigation throughout the application lifecycle; and
Software updates and patching on-demand allows for addressing security vulnerabilities and code flaws.

Four (4) DevSecOps Pillars
The diagram shows that DevSecOps is supported by four pillars: culture, process, technology, and governance.

DevSecOps Lifecycle
The DevSecOps software lifecycle phases. Plan, develop, build, test, release, deliver, deploy, operate, and monitor are the nine (9) phases. Security is implemented in each stage:

The software development lifecycle is not a monolithic linear process with DevSecOps. The Waterfall process’s “big bang” delivery style is replaced with smaller but more frequent deliveries, making it easier to change course as needed. DevSecOps implementation accelerates continuous integration and delivery, and each small delivery is completed through a fully automated or semi-automated process with minimal human intervention. The DevSecOps lifecycle is adaptable and includes numerous feedback loops to ensure continuous improvement.

DevSecOps Framework

The above picture presents a framework for getting started with crucial DevSecOps domains and activities.

DevSecOps Challenges

Issue1: Not Many DevSecOps Talent in the Market
Some continent might facing issue lack of DevSecOps Engineer/Talent since DevSecOps still a broad topic or approach most of organizations

Solution1: Diverify your DevSecOps Team
Instead looking in same region or continental, you have to start looking in differenct region or offshore DevSecOps Engineer to help in your transformation journey.

Issue2: End-of-cycle security testing causes delays and bugs to stay in the production process.
The conventional method of trying to shoehorn security into a nearly finished product is unsatisfactory. Too many flaws are finding their way into production code.

Solution2: Integrate security analysis into the very beginning of the development process.
Instead, security may be woven into the fabric of daily operations and growth. In order to move security compliance into the very early phases of development, developers should be able to do analysis right within the IDE.

Issue3: To "fix" security is a difficult task.
It's a big order for developers and testers who aren't security professionals to be told to "fix" security in the last phases of development when it's unclear what to do.

Solution3: Continually improve the level of security.
There is no need for separate teams to implement their own security rules, such as safe coding standards and static analysis tests for common vulnerabilities.

Issue4: There is no way to know the project's security risk until the last minute.
If a project doesn't have any form of vulnerability assessment, there are no recognised security issues. When late-cycle testing uncovers security flaws, it often leads to rework and further delays. Security flaws still find their way into the hands of users despite these valiant last-minute attempts.

Solution4: Software quality and security are constantly checked.
Compliance reporting or Vulnerability Management tool should give dashboards that are simple to use and provide a high-level and a detailed perspective of the data.

Get Started
It’s a good idea to move to DevSecOps if your firm is currently using DevOps. As a result, DevSecOps is founded on the DevOps mindset, which will make it easier for you to switch. So you may bring together people with diverse strategic backgrounds to work on improving the security procedures that are already in place. This post just discussed the non-technical of DevSecOps transformation; we will dive deeper into the DevSecOps scanner tool and CI/CD implementation using AWS CodePipeline. I hope to see you back in the next post for the DevSecOps CI/CD pipeline and its tools involved.