Forem: 🇦🇺 ☁️ Rumesh Silva ☁️ 🇦🇺

Optimizing AWS ECS: Deregistering and Deleting Unused Task Definition Revisions

🇦🇺 ☁️ Rumesh Silva ☁️ 🇦🇺 — Mon, 11 Dec 2023 11:40:15 +0000

Amazon Web Services (AWS) Elastic Container Service (ECS) offers a robust platform for containerized applications, allowing seamless deployment and scaling. While ECS maintains multiple revisions of task definitions to facilitate rollbacks, managing an excessive amount of unused revisions can become cumbersome over time. In this article, we’ll explore the process of deregistering and deleting unused task definition revisions efficiently using the AWS CLI, especially when dealing with a large number of revisions.

Understanding Task Definition States

Before diving into the deregistration and deletion process, let’s briefly review the different states a task definition revision can be in:

ACTIVE : The task definition revision is currently in use by one or more ECS tasks or services.
INACTIVE : The task definition revision has been deregistered and is no longer in use. It is marked for potential deletion.
DELETE_IN_PROGRESS: Once you’ve initiated the deletion of a task definition, it moves from the INACTIVE state to DELETE_IN_PROGRESS. In this state, Amazon ECS regularly checks if any active tasks or deployments still reference the target task definition. Once it confirms there are none, the task definition is permanently deleted. During the DELETE_IN_PROGRESS state, you’re unable to run new tasks or create new services using that task definition. Importantly, you can initiate the deletion of a task definition at any time without affecting existing tasks and services.

Why Deregister and Delete?

Task definition revisions play a crucial role in versioning and maintaining the history of changes. However, as your application evolves, you may accumulate numerous revisions that are no longer in use. Clearing out these unused revisions not only declutters your ECS environment but also optimizes resource utilization.

Deregistering Task Definition Revisions

Before deleting unused revisions, it’s essential to deregister them. Deregistering makes a task definition revision inactive, marking it for potential deletion.

aws ecs --profile <your_aws_profile> deregister-task-definition --task-definition <your-task-definition-arn>

Deleting Task Definition Revisions

Once you’ve deregistered the revisions, you can proceed with deletion. However, keep in mind that only inactive (deregistered) task definitions can be deleted

aws ecs --profile <your_aws_profile> delete-task-definitions --task-definition <your-task-definition-arn>

Managing Large-Scale Cleanup

While AWS Console provides a graphical interface for managing ECS resources, manually deregistering and deleting task definition revisions becomes impractical when dealing with a high volume of revisions. The AWS CLI proves to be a more efficient solution, especially in scenarios where automation and scripting are crucial.

Scripting : Utilize scripting languages like Bash, Python, or PowerShell to automate the process of deregistering and deleting task definitions. Iterate through a list of inactive revisions and execute the necessary CLI commands or API calls.

Let’s write a simple bash script named deleteandderegistertaskdefs.sh to deregister and delete ECS task definitions using a specified substring/part of the string in the family name;

touch deleteandderegistertaskdefs.sh
chmod u+x deleteandderegistertaskdefs.sh
vi deleteandderegistertaskdefs.sh

#!/bin/bash

# Check if the profile argument is provided
if [ -z "$1" ] || [ -z "$2" ]; then
  echo "Usage: $0 <aws_profile> <substring>"
  exit 1
fi

# AWS profile
AWS_PROFILE="$1"

# Replace with the substring you want to match in the family name
SUBSTRING="$2"

# Function to extract family and revision from a task definition ARN
extract_family_revision() {
  local task_definition_arn="$1"
  local family=$(echo "$task_definition_arn" | awk -F'/' '{print $NF}')
  local revision=$(echo "$family" | awk -F':' '{print $NF}')
  family=$(echo "$family" | awk -F':' '{$NF=""; print $0}' | sed 's/ $//')
  echo "$family:$revision"
}

# Deregister task definitions
deregister_task_definitions() {
  local status="$1"
  local query="taskDefinitionArns[?contains(@, '$SUBSTRING')]"
  local task_definition_arns=$(aws --profile "$AWS_PROFILE" ecs list-task-definitions --status "$status" --query "$query" --output json)

  # Loop through each task definition ARN and initiate their deregistration
  echo "Deregistration of $status Taskdefs has started"
  for task_definition_arn in $(echo "$task_definition_arns" | jq -r '.[]'); do
    family_revision=$(extract_family_revision "$task_definition_arn")

    # Deregister the specific revision of the task definition
    aws --profile "$AWS_PROFILE" ecs deregister-task-definition --task-definition "$family_revision"
  done
  echo "$status Deregistration has finished"
}

# Deregister active task definitions
deregister_task_definitions "ACTIVE"


# Get a list of inactive task definition ARNs matching the family name substring
task_definition_arns=$(aws --profile "$AWS_PROFILE" ecs list-task-definitions --status 'INACTIVE' --query "taskDefinitionArns[?contains(@, '$SUBSTRING')]" --output json)

# Loop through each task definition ARN and initiate their deletion
echo "Deletion has started"
for task_definition_arn in $(echo "$task_definition_arns" | jq -r '.[]'); do
  family_revision=$(extract_family_revision "$task_definition_arn")

  # Deleting the specific revision of the task definition
  aws --profile "$AWS_PROFILE" ecs delete-task-definitions --task-definition "$family_revision"
done
echo "Deletion has finished"

In summary, the script performs the following steps:

Checks for command-line arguments.
Sets AWS profile and substring.
Defines a function to extract family and revision.
Defines a function to deregister task definitions based on status.
Deregisters active task definitions.
Gets a list of inactive task definition ARNs.

Now, you can run the script by providing the AWS profile and substring as command line arguments:

./deleteandderegistertaskdefs.sh profile1 testtaskdef

In this command:

profile1 is the AWS profile argument.
testtaskdef is the substring or part of the string used to filter ECS task definitions.

If you are executing this script within an EC2 instance or application that utilizes an IAM role, it is necessary to modify the code by excluding the profile argument entirely. Run the script as follows:

./deregistertaskdefs.sh testtaskdef

Conclusion

Efficient management of ECS resources, including task definition revisions, is essential for maintaining a streamlined and cost-effective containerized environment. Leveraging the AWS CLI for deregistering and deleting task definition revisions ensures a systematic and automated approach, particularly when dealing with a large number of unused revisions. By incorporating these practices into your ECS maintenance routine, you can optimize resource utilization and keep your containerized applications running smoothly.

Cross-Account Resource Access Using EC2 Instance Metadata.

🇦🇺 ☁️ Rumesh Silva ☁️ 🇦🇺 — Tue, 29 Aug 2023 13:35:36 +0000

In today’s cloud landscape, it’s common for organizations to utilize multiple AWS accounts to manage different aspects of their infrastructure. While maintaining isolation between accounts enhances security, there are scenarios where collaboration across accounts is necessary. Amazon Elastic Compute Cloud (EC2) instances provide a secure way to facilitate cross-account interactions through the use of instance metadata and profiles. This article delves into the process of enabling an EC2 instance in one AWS account to access resources in another AWS account using instance metadata and profiles. We’ll also explore alternative methods for cross-account resource access.

Understanding Cross-Account Access with EC2 Instance Metadata and Profiles

Cross-account resource access involves allowing an entity in one AWS account (the source account) to access resources in another AWS account (the target account). EC2 instance metadata and profiles provide a mechanism to achieve this securely:

**Instance Metadata : **EC2 instance metadata is a service provided by AWS that offers a way to retrieve information about an instance without the need to authenticate. This metadata is stored in a well-known URL, http://169.254.169.254/latest/meta-data/, within the instance itself. By accessing this URL, instances can retrieve details such as instance ID, instance type, security groups, IAM role name, and much more. In addition to providing instance information, EC2 instance metadata is also used to retrieve temporary security credentials. These credentials grant the instance permissions to access resources in the target account. This capability provides a way to grant instances specific permissions without having to embed long-lived credentials directly into the instance.
Profiles : A profile is a collection of settings and credentials that you can use to specify the AWS resources you want to interact with. Further, profiles allow instances to assume cross-account roles using instance metadata without explicitly specifying role ARNs.

Cross-account access using EC2 instance metadata involves the following components:

*Cross-Account IAM Role *:: In the target AWS account, you create an IAM role with the necessary permissions. This role defines which resources the EC2 instance in the source account is allowed to access.
Trust Relationship : The cross-account IAM role must have a trust relationship policy that specifies the AWS account of the source EC2 instance. This trust policy establishes a connection between the two accounts.
Instance Metadata : The EC2 instance in the source AWS account accesses instance metadata to retrieve temporary credentials associated with the cross-account IAM role. These credentials grant the instance access to resources in the target account.

Real World Use Cases

*Data Aggregation *: An EC2 instance in one account could collect and aggregate data from multiple AWS accounts without exposing long-lived credentials.
Centralized Logging : Instances in various accounts can push logs to a central logging bucket in a different account.
Cross-Account Data Processing : Instances in one account can process data stored in buckets owned by other accounts.

Steps to Enable Cross-Account Access

Let’s illustrate the process with a practical example:

Scenario: An EC2 instance in Account A(source account) requires access a particular Route53 hosted zone residing in Account B(target account) for the purpose of creating Route53 records.

) Launch an EC2 Instance in Account A

An EC2 instance initiated within Account A and associated with an IAM Role named “AppServerRole”

AppServerRole Trust Policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "ec2.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

AppServerRole Permission Policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "assumerole",
            "Action": [
                "sts:AssumeRole"
            ],
            "Resource": "arn:aws:iam::<<Account Number of Account B>>:role/<<Role Name>>",
            "Effect": "Allow"
        }
    ]
}

The JSON policy document provided above grants permissions for the AppServerRole to assume a role (sts:AssumeRole) in Account B.

Please substitute <> with the name of the Cross Account Role that will be generated in Step 2.

Retrieving Ec2 Instance Metadata

Retrieving EC2 instance metadata involves accessing information about an Amazon EC2 instance’s configuration and identity. This information is available via a web service running on a special IP address, 169.254.169.254, within the EC2 instance's network.

IMDSv1

curl http://169.254.169.254/latest/meta-data/

he above curl command is used to directly access the EC2 instance metadata service without using IMDSv2 authentication. This approach uses IMDSv1, which can be susceptible to certain security vulnerabilities, especially Server-Side Request Forgery (SSRF) attacks.

IMDSv2

The above curl command is used to securely access the instance metadata service using IMDSv2 authentication. The session token adds an additional layer of security and helps mitigate potential vulnerabilities associated with directly accessing instance metadata using IMDSv1. This approach is recommended for security-sensitive environments and scenarios.

The key differences between Instance Metadata Service Version 1 (IMDSv1) and Instance Metadata Service Version 2 (IMDSv2) in Amazon Web Services (AWS) lie in their security mechanisms and protections. IMDSv2 was introduced to address security concerns and enhance the overall security posture when accessing instance metadata.

2.) Create an IAM Role in Account B (Cross-Account IAM Role)

3.) Attach Policies to Cross-Account IAM Role in Account B

R53Role Trust Policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Statement1",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<<Account Number of Account B>>:role/AppServerRole"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

The above trust policy grants permissions for ‘AppServerRole’ residing in Account A to assume ‘R53Role’ residing in Account B.

R53Role Permission Policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "r53",
            "Effect": "Allow",
            "Action": [
                "route53:GetHostedZone",
                "route53:ChangeResourceRecordSets",
                "route53:GetChange"                
            ],
            "Resource": "arn:aws:route53:::hostedzone/<<hosted-zone-id>>"
        }
    ]
}

The above policy enables permission for specific Route 53 Actions (querying hosted zones, modifying resource record sets, and obtaining change details) to a designated hostedzone in Account B.

4.) Configuring a profile for Amazon EC2 metadata in Account A

Prerequisite : Ensure that the AWS CLI is installed and properly configured on the EC2 instance

Access the EC2 instance through SSH or AWS Systems Manager (SSM). Navigate to the AWS CLI configuration file located at ~/.aws/config and append a new profile, such as "account-B" or a name you prefer. In the profile definition, include the ARN of the role you intend to assume (the ARN of the R53Role residing in Account B). Additionally, set the credential_source as Ec2InstanceMetadata.

[profile account-B]
role_arn = arn:aws:iam::<<Account Number of Account B>>:role/R53Role
credential_source = Ec2InstanceMetadata
region = ap-southeast-2

Above configuration snippet for an AWS CLI profile named “account-B” that assumes an IAM role from Account B using EC2 instance metadata. This profile allows an EC2 instance to assume the specified role in Account B and use the obtained credentials to interact with AWS resources (Route 53 Hosted Zone in Account B).

5.) Verifying Cross Account Access

Connect to the EC2 instance via SSH or AWS Systems Manager (SSM), then execute the following AWS CLI command :

 aws route53 get-hosted-zone --id <<hosted zone id>> --profile account-B

The output above verifies that the EC2 instance located in Account A successfully retrieved details about a hosted zone situated in Account B.

Now, let’s try to perform an action in Account B that lacks the necessary authorization, such as attempting to list hosted zones.

This error message indicates that the AWS identity associated with the role R53Role assumed by the EC2 instance in Account A does not possess the necessary permissions to perform the route53:ListHostedZones action.

Other Cross-Account Access Methods

IAM Role Switching : Instances assume roles directly using temporary credentials from instance metadata. This method requires explicit role switching and doesn’t rely on profiles.
*AWS STS *: Account A requests temporary credentials from STS to assume a cross-account role. This method can be used programmatically to generate credentials. This method can be used for cross-account access beyond EC2 instances.
*Resource Based Access *: Resources like Amazon S3 buckets can be configured to allow access from specific AWS accounts, enabling cross-account resource sharing without temporary credentials.

Conclusion

Leveraging EC2 instance metadata and profiles enables seamless cross-account interactions while maintaining a high level of security. This approach provides the necessary permissions to access resources across accounts without exposing long-lived credentials. In scenarios where security and collaboration are paramount, this mechanism shines.

Other methods like IAM role switching, AWS STS, and resource-based access offer different avenues for cross-account resource sharing, each with its own use cases and considerations. Understanding these methods empowers you to choose the most suitable approach for your specific requirements.

As you embrace cross-account interactions, remember to follow best practices, manage permissions judiciously, and keep security at the forefront of your architecture.

Interactively Accessing Amazon ECS Fargate Containers using AWS Systems Manager Session Manager & ECS Exec

🇦🇺 ☁️ Rumesh Silva ☁️ 🇦🇺 — Thu, 17 Aug 2023 17:36:26 +0000

Amazon ECS (Elastic Container Service) Fargate is a powerful service that allows you to run containers without managing the underlying infrastructure. While Fargate offers numerous benefits in terms of scalability and ease of use, it can sometimes be challenging to interact with containers running within Fargate. AWS Systems Manager Session Manager, combined with ECS Exec, offers a secure and efficient solution for interactively accessing and managing Fargate containers without compromising security or requiring direct SSH access. This guide will walk you through the steps to enable and use ECS Exec with AWS Systems Manager Session Manager to access ECS Fargate containers interactively.

How does ECS Exec function ?

ECS Exec operates by utilizing AWS Systems Manager Session Manager to create and manage secure communication channels between your local machine and the containers running within Amazon ECS Fargate tasks. This architecture ensures a secure, isolated, and interactive experience for debugging and troubleshooting containerized applications.

Prerequisites

For Amazon ECS Exec to work properly, you need to ensure that you meet several prerequisites to set up the necessary environment and permissions. Here’s a list of prerequisites to ensure ECS Exec works as intended:

ECS Cluster and Fargate Tasks
Have an active Amazon ECS cluster running Fargate tasks with the ECS agent version that supports ECS Exec. The ECS agent must be at least version 1.47.0.
IAM Roles and Policies
The ECS task role used by your Fargate tasks needs to have appropriate permissions to interact with AWS Systems Manager Session Manager.
Session manager plugin for AWS CLI
The session manager plugin is an extension for your AWS CLI that facilitates connecting to EC2 instances or AWS Fargate tasks.
Network Configuration
- ECS tasks must be deployed within a Virtual Private Cloud (VPC).
- Ensure that the necessary networking configurations, such as subnets, security groups, and routes, are properly set up to enable communication between ECS Fargate tasks and Systems Manager.

Enabling network communication between ECS Fargate Task and System Manager

Establishing network connectivity between an ECS Fargate task and AWS Systems Manager requires configuring essential networking elements to guarantee seamless communication between these services.

Fargate Task Networking

In Amazon ECS for AWS Fargate, tasks need to use the “awsvpc” network mode, which grants each task its own elastic network interface.If you choose to use this network mode when launching a task or setting up a service, you need to mention which subnets to connect the network interface to and which security groups to use for the network interface.

1.) Fargate tasks placed in public subnets: The task’s elastic network interface should have a public IP address, and there should be a route either directly to the internet or through a NAT gateway that can send internet requests.In this scenario, the Fargate task can readily interact with the AWS Systems Manager service using the public internet.

2.) Fargate tasks placed in private subnets: For a Fargate task in a private subnet and needs to connect with the AWS Systems Manager (SSM) service,it requires either a NAT gateway within the subnet to route requests to the internet or Interface VPC Endpoints specifically configured for the AWS ssm, ec2Messages and ssmmessages services.

Choosing a VPC endpoint offers an array of benefits, including heightened security, privacy, compliance adherence, improved network performance, and enhanced control over data transmission. These factors make it a preferable option over relying on public internet connectivity for interacting with AWS services like AWS Systems Manager.

Setting up ECS Exec for Fargate Tasks

Step 1: Installing Session Manager Plugin for AWS Cli

To install the Session Manager plugin, refer to the AWS documentation and follow the instructions tailored to your client’s operating system.

The following example shows the installation process of the Session Manager plugin on a Mac OS;

Step 2: Include SSM Permissions for SSM in the ECS Task IAM Role

Attach a policy to ECS Task IAM role that grants permissions for ECS Exec, such as the following example policy:

{
  "Effect": "Allow",
  "Action": [
    "ssmmessages:CreateControlChannel",
    "ssmmessages:CreateDataChannel",
    "ssmmessages:OpenControlChannel",
    "ssmmessages:OpenDataChannel"
  ],
  "Resource": "*"
}

Step 3: Incorporate ECS ExecuteCommand permissions into your IAM Role

Make sure that you’ve added the necessary ECS ExecuteCommand permission to your IAM role. Add a policy that grants ECS ExecuteCommand permission. Here’s an example policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ecs:ExecuteCommand"
      ],
      "Resource": "*"
    }
  ]

Step 4: Activate ECS Exec for your services

i.) List the clusters

aws ecs list-clusters --profile profile1

This command will return a list of ECS cluster ARNs (Amazon Resource Names) associated with the specified AWS CLI profile, “profile1”. Make sure you have the necessary credentials and permissions configured in “profile1” to access the ECS service.

ii.) List the Task Definitions

aws ecs list-task-definitions --profile profil

By executing this command, you’ll receive a list of ARNs (Amazon Resource Names) representing the available ECS task definitions in the “profile1” AWS CLI profile.

iii.) List the services within TestCluster

aws ecs list-services --cluster TestCluster  --profile profile1

By executing this command, you’ll receive a list of ARNs (Amazon Resource Names) representing the services associated with the “TestCluster” ECS cluster in the “profile1” AWS CLI profile.

iv) Enable ECS Exec

Enable ECS Exec for an exisiting ECS Service

aws --profile profile1 ecs update-service \
--cluster TestCluster \
--service TestServic2 \
--enable-execute-command \
--force-new-deployment

By running this command, you’re enabling the ECS Exec feature for the specified service and ensuring a new deployment of tasks within the service. This ensures that ECS Exec is effectively integrated into the updated tasks.

Enable ECS Exec for a new ECS Service

aws ecs create-service --cluster TestCluster \
--service-name TestService3 \
--task-definition TestTaskDef:3 \ 
--desired-count 1 \ 
--network-configuration "{\"awsvpcConfiguration\":{\"subnets\":[\"subnet-0b642d3591fe3cf87\"],\"assignPublicIp\":\"ENABLED\"}}" \
--launch-type FARGATE \
--enable-execute-command \
--profile profile1

By executing this command, you’re creating a new ECS service that deploys the specified criteria, including enabling the ECS Exec feature and configuring the network settings (Deploy the Fargate task in a public subnet and activate automatic assignment of public IP addresses for the NICs) for Fargate tasks.

Step 5: Accessing the Container using ECS exec

Upon completing all the aforementioned steps, you will now observe the existence of the following two services running within the cluster.

Let’s proceed to attempt accessing TestService3 using AWS Systems Manager (SSM).

Choose TestService3, navigate to the Tasks tab, and make a note of the Task ID(cf0be9da96e54446984217c9921435ec) and Container Name (TestContainer). Afterward, execute the following command;

aws --profile profile1 ecs execute-command --cluster TestCluster \
--task cf0be9da96e54446984217c9921435ec \                                                                                                         
--container TestContainer \
--command "/bin/sh" \     
--interactive

By running this command, you’re utilizing the “profile1” profile to trigger the execution of the specified command within the ECS task, providing an interactive shell interface for interaction.

Congratulations! You’ve successfully remotely accessed the running container, powered by Fargate. This remote access allows you to troubleshoot any errors with remarkable ease.

How do I troubleshoot errors I receive when performing Amazon ECS Exec on my Fargate tasks?

Conclusion

Amazon ECS Fargate, coupled with AWS Systems Manager Session Manager and ECS Exec, empowers developers and operators to dynamically troubleshoot and manage containers securely. By following this guide, you can efficiently access and interact with Fargate containers, streamline debugging, and ensure the operational success of your containerized applications.

Understanding AWS Scaling: Achieving Efficiency and Resilience in the Cloud

🇦🇺 ☁️ Rumesh Silva ☁️ 🇦🇺 — Thu, 10 Aug 2023 14:45:25 +0000

Introduction

In the ever-changing world of cloud computing, it’s crucial for modern apps to grow when needed. Amazon Web Services (AWS) is a leader in making this happen. They offer ways for businesses to adjust to changing needs without a hitch. This article explains AWS scaling: why it matters, the good things it brings, and the ways it helps apps perform better, cost less, and stay strong.

The Essence of Scaling in AWS:

At its core, scaling in AWS refers to the practice of adjusting the capacity of computing resources — such as virtual machines, containers, databases, or serverless functions — to match the current workload. The goal is to ensure optimal performance and cost efficiency by having the right amount of resources available at the right time. AWS scaling addresses the challenges of both under-provisioning, which can lead to performance issues, and over-provisioning, which results in unnecessary costs.

Why Scaling Matters:

Scaling is crucial for several reasons;

**Performance: **Scaling ensures that applications can handle varying levels of traffic without degradation in performance. It keeps response times fast and maintains a positive user experience.
Cost Efficiency: By dynamically adjusting resources to match the demand, scaling helps prevent over-paying for unused resources. It optimizes resource allocation and cost management.
**Resilience: **Scaling enhances application availability and resilience. When one instance fails, the load can be distributed across other healthy instances, minimizing downtime.
Agility: With the cloud’s elasticity, businesses can quickly respond to changes in demand, whether due to unexpected traffic spikes or seasonal variations.

Scaling Strategies in AWS

AWS Scaling can be categorized broadly into two types: Vertical Scaling and Horizontal Scaling.

Vertical Scaling : Also known as “scaling up” or “resizing,” involves adjusting the size or capacity of an individual instance within a system. It typically entails increasing or decreasing the resources allocated to a single instance, such as its CPU, memory, storage, or network capacity. Vertical Scaling is like upgrading your computer’s hardware to a more powerful model.

When you’re using AWS, vertical scaling means changing the “size” of your virtual machine. It’s like getting a bigger or smaller instance type. For example, you could make your instance go from being small (like a t2.micro) to being bigger (like a t2.large). This helps your instance work better. Vertical Scaling is good when your application needs more resources, but you don’t need to use many instances at once

But vertical scaling has its boundaries. At some point, making things bigger could become too expensive or the kind of instance type you want might not be an option. That’s when horizontal scaling comes in handy — it means adding more computers to help out.

Horizontal Scaling: **Also referred to as “scaling out**,” involves increasing the number of instances or resources to handle a larger load or to improve system redundancy. Instead of making individual instances more powerful (as in vertical scaling), horizontal scaling adds more instances to distribute the workload across multiple resources.

In AWS, when it comes to horizontal scaling, it usually means putting more instances into a group that can automatically adjust the number of instances as needed. This helps the system handle more users or tasks without relying only on one big instance.

Horizontal Scaling works best for applications that can be divided and spread out over many instances. This helps if one instance stops working, as the others can still handle the work. But keep in mind that handling lots of instances might need extra setup and automation to use them effectively.

Further Breakdown of AWS Scaling

What is Auto-Scaling ?

AWS Auto Scaling is a service provided by Amazon Web Services that automatically adjusts the number of resources, such as Amazon EC2 instances or ECS tasks, in a group to match the desired performance, availability, and cost requirements. It helps ensure that your application can handle varying levels of traffic without manual intervention.

With AWS Auto Scaling, you define the desired number of instances or other resources that you want to maintain, and the service automatically scales the group up or down based on factors such as CPU utilization, network traffic, or custom metrics you define. This dynamic scaling ensures that your application remains responsive and cost-efficient, as it can automatically add resources during peak demand and remove them during periods of lower activity.

In addition to maintaining consistent performance, AWS Auto Scaling also enhances application availability and reduces the risk of overprovisioning or underprovisioning resources. This service is particularly useful in scenarios where the workload is unpredictable or experiences fluctuations over time.

Horizontal Scaling Strategies

**Predictive Scaling : *Predictive Scaling in AWS is an scaling approach that uses historical data and machine learning algorithms to forecast future traffic patterns. This enables the system to proactively adjust resources before the expected surge or drop in demand occurs. By analyzing past usage trends, predictive scaling ensures that the right number of resources is available precisely when needed, optimizing performance and cost-efficiency.
**Example* : Imagine you run an online store, and you know that during holidays like Black Friday, your website gets a lot more visitors. With Predictive Scaling, AWS can analyze past holiday seasons’ data and predict when your website will have the most visitors. It will automatically add more servers before the rush starts, ensuring your website doesn’t slow down or crash during the high traffic times.
Scheduled Scaling : Resources are adjusted based on a predefined schedule, useful for predictable demand changes, such as peak business hours.
Example : Let’s say you have a web application that experiences predictable changes in traffic throughout the day. During business hours, the number of users accessing your application increases, but at night, the usage drops significantly. Instead of keeping the same number of servers running all the time, which could be wasteful and expensive, you can use Scheduled Scaling.
Manual Scaling **: This involves adjusting resources based on manual intervention. It offers more control but may not be as responsive to rapidly changing workloads.
**Example: Suppose you run an online store, and you’re running a special promotion that you expect will bring a surge of visitors to your website. To make sure your website doesn’t slow down or crash during this busy period, you can use Manual Scaling.
Dynamic Scaling : This approach automatically adjusts resources based on real-time workload changes. AWS Auto Scaling is a service that embodies this strategy, allowing you to define scaling policies based on metrics such as CPU utilization or request count.
Example : Imagine you have a mobile app that offers real-time updates during a live sports event. Normally, your app has a steady number of users, but during the match, the traffic can spike significantly. This is where Dynamic Scaling comes in.

AWS Dynamic Scaling Policies

AWS Dynamic Scaling Policies are rules or instructions that tell AWS how to automatically adjust the number of resources, such as instances or containers, based on real-time conditions. These policies help ensure that your applications are responsive, efficient, and cost-effective.

Following are different types of dynamic scaling policies in AWS, each addressing specific scenarios:

Simple Scaling : With this policy, you define specific thresholds for a metric, such as CPU usage. When the metric goes beyond these thresholds, AWS adds or removes resources as needed.
Simple Scaling Policies are exclusively available within Ec2-AutoScaling.
Target Tracking Scaling : This policy keeps a specific metric, like CPU utilization or request rate, at a target value. AWS automatically adds or removes resources to maintain this target, adapting to changing demand.
Both EC2 Auto Scaling and Application Auto Scaling provide support for Target Tracking Policies.
Step Scaling : Step Scaling allows you to set up scaling adjustments at specific intervals or steps. For instance, you might add more resources if a metric reaches a certain level and then add even more if it exceeds another threshold.
Both EC2 Auto Scaling and Application Auto Scaling provide support for Step Scaling Policies.
Schedule Scaling : This policy lets you plan scaling actions based on a schedule. You can increase resources before expected traffic spikes and decrease them during quieter times.
Schedule Scaling Policies are exclusively available within Application-Auto Scaling.

Conclusion:

AWS scaling serves as the backbone of a resilient, high-performance, and cost-effective cloud infrastructure. It empowers businesses to dynamically adapt to varying workloads while optimizing resource allocation and minimizing operational complexities. By implementing appropriate scaling strategies and leveraging AWS’s scaling services, organizations can future-proof their applications and ensure exceptional user experiences in the ever-evolving landscape of cloud computing.

Associating a VPC in a Different AWS Account with a Hosted Zone

🇦🇺 ☁️ Rumesh Silva ☁️ 🇦🇺 — Fri, 04 Aug 2023 04:12:25 +0000

Introduction

Amazon Web Services (AWS) provides a robust environment for building and managing cloud infrastructure. However, certain scenarios require the association of resources across different AWS accounts. One such scenario involves associating a Virtual Private Cloud (VPC) located in one AWS account with a Route 53 hosted zone located in another AWS account. While the AWS Management Console facilitates VPC-to-hosted zone association within the same account, cross-account association requires a different approach. In this article, we will explore how to achieve this association using AWS Command Line Interface (CLI) commands.

Use-case

The use case of associating a VPC in a different AWS account with a hosted zone involves enabling resources in one AWS account’s Virtual Private Cloud (VPC) to interact with a hosted zone in another AWS account’s Route 53 service. This association is useful for DNS resolution and communication purposes.

For the purpose of illustration, consider the following scenario where Account_B requires DNS resolution for the private hosted zone in Account_A.

Two AWS accounts, referred to as Account_A and Account_B, with corresponding account numbers 11111111 and 22222222, respectively.
A Private Hosted Zone has been established in Account_A, with the Hosted Zone ID being Z458514111102.
In Account_B, there exists a VPC identified by the VPC ID vpc-1458522bhuf.
You’ve configured two AWS profiles on your local computer, each assuming the corresponding AWS role with Route 53 permissions in the respective target accounts. The profiles are as follows:
- Account_A is represented by profile-A, and Account_B is represented by profile-B.

Profile_A Access Permissions:

List and get hosted zone in Route 53: route53:Get, route53:List**
Create and manage hosted zones in Route 53: *Route53:*HostedZone*
Create and manage VPC association authorizations : route53:*VPCAssociationAuthorization*

Profile_B Access Permissions:

Permissions to associate a VPC with a hosted zone : route53:AssociateVPCWithHostedZone
List and describe VPCs in Account_B : ec2:DescribeVpcs

Let’s examine the steps required to associate the VPC in Account_B with the hosted zone in Account_A.

Step 1: Create an association-authorization request in Account_A, the account where the hosted zone resides.

Following command should be executed in the account where the zone is intended to be shared, it is Account_A in our scenario.

aws route53 create-vpc-association-authorization --hosted-zone-id Z458514111102 --vpc VPCRegion=ap-southeast-2,VPCId=vpc-1458522bhuf --profile profile-A

This AWS CLI command initiates the process of creating an association-authorization request in Account_A. This request allows the VPC (vpc-1458522bhuf) from Account_B to be associated with the hosted zone specified by its ID (Z458514111102) in Account_A. The action is performed using the profile-A credentials for authentication.

Step 2: Associate the VPC in Account_B with the hosted zone in Account_A

*Following command should be executed inthe account that requires access to the private zone using AWS Route 53, *In our scenario, it pertains to Account_B.

 aws route53 associate-vpc-with-hosted-zone --hosted-zone-id Z458514111102 --vpc VPCRegion=ap-southeast-2,VPCId=vpc-1458522bhuf --profile profile-B

This command performs the association of the specified VPC (vpc-1458522bhuf) from Account_B with the hosted zone identified by its ID (Z458514111102) in Account_A. The process takes place using the **profile-B **credentials for authentication.

Upon completing the aforementioned two steps, the VPC located in Account_B has been effectively associated with the private hosted zone in Account_A.

Let’s confirm the above by executing the following command;

aws route53 list-hosted-zones-by-vpc --vpc-id vpc-1458522bhuf --vpc-region ap-southeast-2 --profile profile-B

This command will provide you with information about the all the private hosted zones that the VPC in Account_B (vpc-1458522bhuf) is associated with

This can be also confirmed using the Route 53 console in Account_A.

What is the outcome of the above ?

The outcome of the above process is the successful establishment of an association between the VPC in Account_B and the private hosted zone in Account_A. This means that any DNS* queries originating from resources within the VPC in *Account_B** will be able to resolve records from the associated private hosted zone in Account_A. This enables seamless communication and resource access between the VPCs in different AWS accounts using the DNS names defined in the*** private hosted zone***.

Step 3: Delete association-authorization request initiated in Step 1 (recommended).

Following commands should be executed in the account where the Association Authorization request is created , it is Account_A in our scenario

List the Authorizations created in Account _A

aws route53 list-vpc-association-authorizations --hosted-zone-id Z458514111102 --profile profile-A

Delete the VPC Authorization Association request

aws route53 delete-vpc-association-authorization --hosted-zone-id Z458514111102 --vpc VPCRegion=ap-southeast-2,VPCId=vpc-1458522bhuf --profile profile-A

This command removes the association-authorization request in Account_A that allowed the VPC (vpc-1458522bhuf) from Account_B to be associated with the hosted zone specified by its ID (Z458514111102) in Account_A. The action is performed using the profile-A credentials for authentication.

Deleting the associations is part of proper resource management. It helps you keep your AWS environment organized and efficient by removing unnecessary permissions.

Remember that deleting the association-authorization request won’t impact the existing associations between the VPC and the hosted zone. It simply prevents new associations from being made.

Conclusion

While the AWS Management Console provides an intuitive interface for many AWS tasks, certain scenarios, such as associating a VPC from one account with a Route 53 hosted zone in another account, require the power and flexibility of the AWS CLI. By following this guide, you can successfully accomplish cross-account VPC associations, ensuring efficient resource management and improved security across your AWS infrastructure.

A Comprehensive Guide to Various Sceptre Commands

🇦🇺 ☁️ Rumesh Silva ☁️ 🇦🇺 — Thu, 03 Aug 2023 00:43:02 +0000

Introduction

Sceptre provides various commands that cater to different aspects of managing CloudFormation stacks and interacting with AWS resources. In this article, we’ll explore the various Sceptre commands and how they can streamline your cloud infrastructure management.

Sceptre CLI (Version 4.2.2)

Please consult the official Sceptre documentation or the documentation specific to version 4.2.2 for the most accurate and up-to-date information on commands and their usage.

Usage

Usage: sceptre [OPTIONS] COMMAND [ARGS]...

Sceptre serves as a command-line tool, and if you run it without a sub-command, it will display helpful information by showing a list of the available commands.

sceptre 
sceptre --help

Various Sceptre Commands

1.) create



    sceptre create --help
    Usage: sceptre create [OPTIONS] PATH [CHANGE_SET_NAME]

      Creates a stack for a given config PATH. Or if CHANGE_SET_NAME is specified
      creates a change set for stack in PATH.

    Options:
      -y, --yes                       Assume yes to all questions.
      --disable-rollback / --enable-rollback
                                      Disable or enable the cloudformation
                                      automatic rollback
      --help                          Show this message and exit.

Create a stack

usage: sceptre create [options] PATH

sceptre create -y s3-bucket-config.yaml

Create a changeset

Using changesets is a best practice when managing CloudFormation stacks, especially in production and other controlled environments. It promotes a well-defined and cautious approach to making changes, reducing the risk of disruptions and ensuring the stability of your cloud infrastructure.

For additional details on changesets, please check this page.

usage: sceptre create [options] PATH [CHANGE_SET_NAME]

sceptre create -y --enable-rollback s3-bucket-config.yaml updatename

The command creates changeset called “updatename” stack based on the provided configuration file “s3-bucket-config.yaml.” The “-y” option skips confirmation prompts, and “ — enable-rollback” enables automatic rollback in case of stack creation failures.

2.) delete

sceptre delete --help
Usage: sceptre delete [OPTIONS] PATH [CHANGE_SET_NAME]

  Deletes a stack for a given config PATH. Or if CHANGE_SET_NAME is specified
  deletes a change set for stack in PATH.

Options:
  -y, --yes  Assume yes to all questions.
  --help     Show this message and exit.

Delete a stack

usage: sceptre delete [options] PATH

sceptre delete -y s3-bucket-config.yaml

Delete a changeset

usage: sceptre delete [options] PATH [CHANGE_SET_NAME]

sceptre delete -y s3-bucket-config.yaml updatename

3.) describe

sceptre describe --help
Usage: sceptre describe [OPTIONS] COMMAND [ARGS]...

  Commands for describing attributes of stacks.

Options:
  --help  Show this message and exit.

Commands:
  change-set  Describes the change set.
  policy      Displays the stack policy used.

 sceptre describe change-set  s3-bucket-config.yaml updatename

The above output suggests that the change set “updatename” includes a modification to the existing AWS S3 bucket resource named “MyS3Bucket” within the “my-s3-bucket-stack” CloudFormation stack. The change involves updating the properties of the bucket. Additionally, the resource is marked for **replacement, indicating that a new physical resource will be created to apply the update.

Change sets can also be examined through the AWS CloudFormation console, providing a user-friendly graphical interface to visualize the alterations made to stack resources.

 sceptre describe policy  s3-bucket-config.yaml

4.) diff

sceptre diff --help               
Usage: sceptre diff [OPTIONS] PATH

  Indicates the difference between the currently DEPLOYED stacks in the
  command path and the stacks configured in Sceptre right now. This command
  will compare both the templates as well as the subset of stack
  configurations that can be compared. By default, only stacks that would be
  launched via the launch command will be diffed, but you can diff ALL stacks
  relevant to the passed command path if you pass the --all flag.

  Some settings (such as sceptre_user_data) are not available in a
  CloudFormation stack description, so the diff will not be indicated.
  Currently compared stack configurations are:

    * parameters
    * notifications
    * cloudformation_service_role
    * stack_tags

  Important: There are resolvers (notably !stack_output) that rely on other
  stacks to be already deployed when they are resolved. When producing a diff
  on Stack Configs that have such resolvers that point to non-deployed stacks,
  this presents a challenge, since this means those resolvers cannot be
  resolved. This particularly applies to stack parameters and when a stack's
  template uses sceptre_user_data with resolvers in it. In order to continue
  to be useful when producing a diff in these conditions, this command will do
  the following:

  1. If the resolver CAN be resolved, it will be resolved and the resolved
  value will be in the diff results. 2. If the resolver CANNOT be resolved, it
  will be replaced with a string that represents the resolver and its
  arguments. For example: !stack_output my_stack.yaml::MyOutput will resolve
  in the parameters to "{ !StackOutput(my_stack.yaml::MyOutput) }".

  Particularly in cases where the replaced value doesn't work in the template
  as the template logic requires and causes an error, there is nothing further
  Sceptre can do and diffing will fail.

Options:
  -t, --type [deepdiff|difflib]  The type of differ to use. Use "deepdiff" for
                                 recursive key/value comparison. "difflib"
                                 produces a more traditional "diff" result.
                                 Defaults to deepdiff.
  -s, --show-no-echo             If set, will display the unmasked values of
                                 NoEcho parameters generated LOCALLY (NoEcho
                                 parameters for deployed stacks will always be
                                 masked when retrieved from CloudFormation.).
                                 If not set (the default), parameters
                                 identified as NoEcho on the local template
                                 will be masked when presented in the diff.
  -n, --no-placeholders          If set, no placeholder values will be
                                 supplied for resolvers that cannot be
                                 resolved.
  -a, --all                      If set, will perform diffing on ALL stacks,
                                 including ignored and obsolete ones;
                                 Otherwise, it will diff only stacks that
                                 would be created or updated when running the
                                 launch command.
  --help                         Show this message and exit.

sceptre diff s3-bucket-config.yaml

The detected difference reveals that the “bucketname” parameter within the CloudFormation stack has been modified, changing from “first-secptre-bucket-20230728” to “first-secptre-bucket-20230729.” Notably, the CloudFormation template itself remains unchanged during this update.

5.) drift

A “drift” refers to a situation where the actual state of a stack’s resources deviates from the expected state defined in the CloudFormation template. In other words, a drift occurs when there are resource changes made directly in the AWS environment, outside of CloudFormation’s control(via Sceptre).

sceptre drift --help
Usage: sceptre drift [OPTIONS] COMMAND [ARGS]...

  Commands for calling drift detection.

Options:
  --help  Show this message and exit.

Commands:
  detect  Run detect stack drift on running stacks.
  show    Shows stack drift on running stacks.

detect drift

 sceptre drift detect s3-bucket-config.yaml

The above output indicates that the “my-s3-bucket-stack” CloudFormation stack is in good condition and does not have any drifted resources. The drift detection process has been completed, and all the resources in the stack are in sync with the defined CloudFormation template.

show drift

 sceptre drift show s3-bucket-config.yaml

The output indicates that the specific AWS S3 Bucket resource with the logical ID “MyS3Bucket” within the “my-s3-bucket-stack” CloudFormation stack is in sync with the expected state defined in the template. There are no property differences, and the resource’s properties match those defined in the CloudFormation template, ensuring that it is in the desired state.

6.) dump

 sceptre dump --help
Usage: sceptre dump [OPTIONS] COMMAND [ARGS]...

  Commands for dumping attributes of stacks.

Options:
  --help  Show this message and exit.

Commands:
  all       Dumps both the rendered (post-Jinja) Stack Configs and the...
  config    Dump the rendered (post-Jinja) Stack Configs.
  template  Prints the template used for stack in PATH.

sceptre dump template s3-bucket-config.yaml

sceptre dump config  s3-bucket-config.yaml

sceptre dump all  s3-bucket-config.yaml

7.) estimate-cost

sceptre estimate-cost --help
Usage: sceptre estimate-cost [OPTIONS] PATH

  Prints a URI to STOUT that provides an estimated cost based on the resources
  in the stack. This command will also attempt to open a web browser with the
  returned URI.

Options:
  --help  Show this message and exit.

sceptre estimate-cost  s3-bucket-config.yaml

8.) execute

sceptre execute --help
Usage: sceptre execute [OPTIONS] PATH CHANGE_SET_NAME

  Executes a Change Set.

Options:
  -y, --yes                       Assume yes to all questions.
  --disable-rollback / --enable-rollback
                                  Disable or enable the cloudformation
                                  automatic rollback
  --help                          Show this message and exit.

sceptre execute -y --enable-rollback s3-bucket-config.yaml updatename

Prior to executing the changeset, it’s advisable to thoroughly review its contents. As demonstrated in the ‘describe changeset’ command above, taking this precaution is essential because the changes applied by the changeset can be irreversible and may not have a straightforward rollback mechanism.

As per the above output;

The AWS S3 bucket “MyS3Bucket” within the “s3-bucket-config” stack is being updated.
Since the bucket update requires the creation of a new physical resource(as the bucket name has been modified) , it indicates that CloudFormation is replacing the existing bucket with a new one. This suggests that the “Delete” deletion policy is applied to the bucket and the new bucket is created, and the old one is deleted.

9.) fetch-remote-template

sceptre fetch-remote-template --help
Usage: sceptre fetch-remote-template [OPTIONS] PATH

  Prints the remote template used for stack in PATH.

Options:
  --help  Show this message and exit.

 sceptre fetch-remote-template s3-bucket-config.yaml

10.) generate

sceptre generate --help
Usage: sceptre generate [OPTIONS] PATH

  Prints the template used for stack in PATH.

  This command is aliased to the dump template command for legacy support
  reasons. It's the same as running `sceptre dump template`.

Options:
  -n, --no-placeholders  If True, no placeholder values will be supplied for
                         resolvers that cannot be resolved.
  --help                 Show this message and exit.

 sceptre generate s3-bucket-config.yaml

11.) launch

sceptre launch --help                 
Usage: sceptre launch [OPTIONS] PATH

  Launch a Stack or StackGroup for a given config PATH. This command is
  intended as a catch-all command that will apply any changes from Stack
  Configs indicated via the path.

  * Any Stacks that do not exist will be created
  * Any stacks that already exist will be updated (if there are any changes)
  * If any stacks are marked with "ignore: True", those stacks will neither be created nor updated
  * If any stacks are marked with "obsolete: True", those stacks will neither be created nor updated.
  * Furthermore, if the "-p"/"--prune" flag is used, these stacks will be deleted prior to any
    other launch commands

Options:
  -y, --yes                       Assume yes to all questions.
  -p, --prune                     If set, will delete all stacks in the
                                  command path marked as obsolete.
  --disable-rollback / --enable-rollback
                                  Disable or enable the cloudformation
                                  automatic rollback
  --help                          Show this message and exit.\

sceptre launch -y s3-bucket-config.yaml

According to the provided output, since the stack is already present, the “launch” procedure has brought about modifications to the resources (specifically, the S3 bucket) within the stack. These alterations align with the adjustments introduced in the CloudFormation template or the configuration of the stack (in our case, the stack configuration has been modified to update the bucket name).

12.) list

sceptre list --help
Usage: sceptre list [OPTIONS] COMMAND [ARGS]...

  Commands for listing attributes of stacks.

Options:
  --help  Show this message and exit.

Commands:
  change-sets  List change sets for stack.
  outputs      List outputs for stack.
  resources    List resources for stack or stack_group.
  stacks       List sceptre stack config attributes,

sceptre list change-sets s3-bucket-config.yaml

sceptre list outputs  s3-bucket-config.yaml

sceptre list resources  s3-bucket-config.yaml

sceptre list stacks  s3-bucket-config.yaml

13.) new

 sceptre new --help
Usage: sceptre new [OPTIONS] COMMAND [ARGS]...

  Commands for initialising Sceptre projects.

Options:
  --help  Show this message and exit.

Commands:
  group    Creates a new Stack Group directory in a project.
  project  Creates a new project.

sceptre new command has been comprehensively explained with an example in a previous article within the section titled “Setting Up the Directory Structure for a New Sceptre Project.”

14.). prune

sceptre prune --help
Usage: sceptre prune [OPTIONS] [PATH]

  This command deletes all obsolete stacks in the project. Only obsolete
  stacks can be deleted via prune; If any non-obsolete stacks depend on
  obsolete stacks, an error will be raised and this command will fail.

Options:
  -y, --yes  Assume yes to all questions.
  --help     Show this message and exit.

 sceptre prune s3-bucket-config.yaml

How to make a stack “obselete” ? By setting the obsolete parameter to True, you are indicating that this stack is no longer actively managed and is considered obsolete. This helps communicate the status of the stack to the team, making it clear that this stack is not intended for further updates or maintenance.

template:
  path: s3-bucket-template.yaml
  type: file

stack_name: my-s3-bucket-stack
obsolete: True

parameters:
  bucketname: first-secptre-bucket-20230730
  deletionpolicy: Delete

What will be the outcome if you execute the prune command at this moment? The stack will be deleted as it is marked as “obselete”

15.) set-policy

sceptre set-policy --help
Usage: sceptre set-policy [OPTIONS] PATH [POLICY_FILE]

  Sets a specific Stack policy for either a file or using a built-in policy.

Options:
  -b, --built-in [deny-all|allow-all]
                                  Specify a built in stack policy.
  --help                          Show this message and exit.Specifies the resources you wish to safeguard against accidental modifications during a stack update

sceptre set-policy -b allow-all  s3-bucket-config.yaml

The purpose of using this command is to establish a standardized policy that governs what types of changes can be made to the stack resources. The “allow-all” policy, as implied by its name, allows all possible updates to the stack. This can be useful in scenarios where you want to enable unrestricted updates to the stack resources.

Let’s run the describe policy command and see the output now;

Let’s define a custom stack policy in JSON format that you can use to deny updates to all resources within a CloudFormation stack:

config/policies/deny-policy.json

{
    "Statement" : [
      {
        "Effect" : "Deny",
        "Action" : "Update:*",
        "Principal": "*",
        "Resource" : "*"
      }
    ]
  }

sceptre set-policy s3-bucket-config.yaml config/policies/deny-policy.json

Let’s evaluate the impact of the “deny” stack policy on the stack by trying to perform a stack update.

As anticipated, the update operation failed due to the stack policy that prohibits any updates on all resources.

Remember that creating and applying policies should be done carefully, as they significantly impact the actions that can be performed on your stack resources. Always test policies in a controlled environment before applying them to production stacks

16.) update

sceptre update --help 
Usage: sceptre update [OPTIONS] PATH

  Updates a stack for a given config PATH. Or perform an update via change-set
  when the change-set flag is set.

Options:
  -c, --change-set                Create a change set before updating.
  -v, --verbose                   Display verbose output.
  -y, --yes                       Assume yes to all questions.
  --disable-rollback / --enable-rollback
                                  Disable or enable the cloudformation
                                  automatic rollback
  --help                          Show this message and exit.

sceptre update s3-bucket-config.yaml

Updating stack with a changeset

sceptre update --change-set s3-bucket-config.yaml

It’s worth observing that while “Update Change Set” and “Create Change Set” might appear similar, they actually serve distinct purposes. Despite their eventual outcomes being similar, these two operations are applied in different scenarios. “Create Change Set” is typically used when making significant changes to an existing stack, while “Update Change Set” is used for incremental changes to an existing stack. Both operations provide a safety net by allowing you to review changes before they are applied, reducing the risk of unintended consequences.

17.) validate

sceptre validate --help
Usage: sceptre validate [OPTIONS] PATH

  Validates the template used for stack in PATH.

Options:
  -n, --no-placeholders  If True, no placeholder values will be supplied for
                         resolvers that cannot be resolved.
  --help                 Show this message and exit.

sceptre validate s3-bucket-config.yaml

Conclusion

As we explored various Sceptre commands, we learned how they help create, update, and manage cloud infrastructure effortlessly. With its command-line toolkit, Sceptre becomes a dependable companion for modern cloud enthusiasts, making cloud deployment efficient and automated.

Whether you’re experienced in cloud engineering or new to Infrastructure as Code, learning Sceptre commands will surely boost your cloud management skills and speed up your path to cloud expertise. Embrace Sceptre’s capabilities and enter a new phase of managing cloud infrastructure.

By understanding and utilizing these Sceptre commands, you’ll be well-equipped to optimize your cloud infrastructure and ensure the scalability, reliability, and security of your applications.

(Note: This article offers a broad look at Sceptre commands and what they can do. For more detailed and up-to-date information, readers are advised to consult the official Sceptre documentation and additional resources.)

Simplifying Infrastructure Deployment with Sceptre: Your First Stack — An S3 Bucket

🇦🇺 ☁️ Rumesh Silva ☁️ 🇦🇺 — Tue, 01 Aug 2023 00:28:57 +0000

Introduction

In this article, we’ll walk you through creating and deploying your first stack using Sceptre, with a simple example of creating an S3 bucket.

Prerequisites

Before we begin, make sure you have gone through the article on A Guide to Installing and Configuring Sceptre

Building Infrastructure — The S3 Bucket

The Directory Structure

1. Defining the S3 Bucket Config — s3-bucket-config.yaml

 template:
      path: s3-bucket-template.yaml
      type: file

    stack_name: my-s3-bucket-stack

    parameters:
      bucketname: first-secptre-bucket-20230727
      deletionpolicy: Delete

This Sceptre config file creates an S3 bucket using the specified CloudFormation template file (s3-bucket-template.yaml) with the given parameters. The S3 bucket's name will be "my-first-sceptre-bucket-20230727," and it will have the "Delete" deletion policy.

Below is the explanation of each section in the config file:

path: s3-bucket-template.yaml: This line specifies the path to the CloudFormation template file (s3-bucket-template.yaml) relative to the Sceptre configuration file’s location. The template file contains the CloudFormation code that defines the S3 bucket’s properties. The path property may consist of either an absolute or relative path to the template file. When using a relative path, this handler assumes it is relative to the ‘sceptre_project_dir/templates’ directory
type: file: This indicates the type of the template, which in this case is a file. This is one of the possible values for the type property, and it is used to indicate the external source location of the template.(e.g., file,s3,http).

In the provided example, the CloudFormation template is stored in an S3 bucket, and its path is provided using the path attribute. Sceptre will retrieve the template from the specified S3 bucket and use it to create the stack.

template:
  path: s3://my-bucket/s3-bucket-template.yaml
  type: s3

stack_name: my-s3-bucket-stack: This line defines the name of the CloudFormation stack that will be created. In this case, the stack will be named “my-s3-bucket-stack.”
parameters: This section defines the input parameters that will be passed to the CloudFormation stack during its creation. These parameters customize the properties of the S3 bucket being created. The parameters listed are:
– bucketname: This parameter is set to “my-first-sceptre-bucket- 20230727,” which specifies the desired name for the S3 bucket.
– deletionpolicy: This parameter is set to “Delete,” which indicates the desired deletion policy for the S3 bucket. In this case, the bucket will be deleted when the CloudFormation stack is deleted

2. Defining the S3 Bucket Template— s3-bucket-template.yaml

Description: Template creates an empty S3 Bucket

    Parameters:
      bucketname:
        Type: String

      deletionpolicy:
        Type: String
        Default : Retain
        AllowedValues:
          - Delete
          - Retain

    Resources:
      MyS3Bucket: 
        DeletionPolicy: !Ref deletionpolicy
        Type: 'AWS::S3::Bucket'
        Properties:
          BucketName: !Ref bucketname

    Outputs: 
      S3BucketName: 
        Value: !Ref MyS3Bucket 
        Description: Name of the S3 Bucket 
      S3BucketARN: 
        Value: !GetAtt MyS3Bucket.Arn 
        Description: ARN of the Bucket

The above template allows users to create an S3 bucket with a custom name and specify the desired deletion policy. By using the outputs, users can easily access and utilize the bucket name and ARN for further operations and integration within their AWS environment.

Let’s break down each section of the template:

Parameters : This section defines the input parameters that can be provided when launching the CloudFormation stack. In this template, there are two parameters.
Resources : This section defines the AWS resources that the CloudFormation stack will create. In this template, there is one resource. How to specify an S3 bucket in CloudFormation
Outputs : This section defines the information that will be displayed once the CloudFormation stack is created. In this template, there are two outputs.

3. Different Sceptre commands for deploying and managing the stacks

Before executing the Sceptre commands, ensure that you change the current working directory to the project directory. In our scenario, the project directory is named “first-sceptre-directory.”

Validating the stack : You can verify that your CloudFormation templates are well-formed, follow the correct YAML or JSON syntax, and reference valid AWS resources and properties.

sceptre validate s3-bucket-config.yaml

Using Docker

docker run -v "$(pwd):/project"  -v "$HOME/.aws:/root/.aws" -w /project sceptre-image validate s3-bucket-config.yaml

This Docker command creates a container based on the “sceptre-image” Docker image, maps the current working directory and AWS configuration from the host to appropriate directories inside the container, sets the working directory for the Sceptre command to “/project,” and then runs the Sceptre “validate” command on the “s3-bucket-config.yaml” configuration file.

Creating the stack: This command will initiate the stack creation process using the associated CloudFormation template.
sceptre validate s3-bucket-config.yaml

sceptre create s3-bucket-config.yaml

Using Docker

docker run -v "$(pwd):/project"  -v "$HOME/.aws:/root/.aws" -w /project sceptre-image create s3-bucket-config.yaml

Updating the stack: This command will trigger the update process, and CloudFormation will apply the changes specified in the template to the existing stack.

sceptre update s3-bucket-config.yaml

Using Docker

docker run -v "$(pwd):/project"  -v "$HOME/.aws:/root/.aws" -w /project sceptre-image update s3-bucket-config.yaml

After the deployment is successful, go to the AWS Cloudformation console and verify the creation of your stack and S3 bucket with the specified name (my-first-sceptre-bucket in our example).

Deleting the stack: This command will delete the CloudFormation stack and the associated resources.

sceptre delete s3-bucket-config.yaml

Using Docker

docker run -v "$(pwd):/project"  -v "$HOME/.aws:/root/.aws" -w /project sceptre-image delete s3-bucket-config.yaml

Conclusion

Congratulations! You have successfully created and deployed your first stack using Sceptre.

This simple example of creating an S3 bucket serves as a foundation for more complex deployments and scenarios. As you explore additional features and customization options, refer to the official Sceptre documentation for a deeper understanding.

Leveraging Sceptre, you can streamline infrastructure management and elevate your AWS cloud deployment experience. Happy deploying!

A Guide to Installing and Configuring Sceptre

🇦🇺 ☁️ Rumesh Silva ☁️ 🇦🇺 — Thu, 27 Jul 2023 00:32:47 +0000

Sceptre, an open-source tool developed by Cloudreach, offers a robust solution for managing cloud infrastructure as code. In this article, we will explore various methods of installing and configuring Sceptre, including using Docker, based on the official Sceptre documentation.

Installation

Installing Sceptre via pip (Python Package Manager)

The simplest way to install Sceptre is using pip, the Python package manager. First, ensure you have Python installed on your system

Prerequisites

AWS Account: You will need an AWS account to work with Sceptre since it interacts with CloudFormation and other AWS services.
Python: Sceptre requires Python, so ensure you have Python (version 3.6+) installed on your system.

Now that we have the prerequisites covered, let’s proceed with installing Sceptre:

Open your terminal or command prompt.
Use pip to install Sceptre by running the following command:

pip install sceptre

Wait for the installation to complete, and you’re now ready to use Sceptre.
Once the installation is complete, you can verify it by running

sceptre --version

Configuring Sceptre

With Sceptre installed, the next step is to set up the necessary configuration:

Configuring AWS Credentials for Sceptre

Sceptre uses the standard AWS SDK for Python (Boto3) to interact with AWS services, and it follows the same credential resolution order as Boto3. The AWS credentials can be specified in multiple ways, and Sceptre automatically picks up the appropriate credentials based on this order:

i.) Environment Variables: Sceptre checks for the presence of the following environment variables:

AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY : Specifies the AWS access key ID and secret access key.
AWS_SESSION_TOKEN: Specifies the session token for temporary security credentials when using AWS Identity and Access Management (IAM) roles.

ii.) AWS Config File: Sceptre looks for the AWS configuration file (~/.aws/config) that can define multiple profiles, each with its set of credentials. The [default] profile is used if no specific profile is specified.

iii.) AWS Credentials File: Sceptre checks for the AWS credentials file (~/.aws/credentials), which can also define multiple profiles with their respective credentials.

iv.) IAM Role for Amazon EC2 Instance: If Sceptre is running on an Amazon EC2 instance, it automatically retrieves the credentials associated with the IAM role attached to that EC2 instance.

v.) IAM Role via Instance Metadata Service: If Sceptre is running on an Amazon EC2 instance and there is no IAM role directly attached to the instance, it will check the instance metadata service to determine if there is an IAM role associated with the EC2 instance.

vi.) AWS CLI Configuration: If you have configured the AWS CLI with aws configure, Sceptre will also pick up the credentials from the CLI configuration.

vii) Explicitly Defined Credentials: You can explicitly define credentials in your Sceptre configuration files (config/*.yaml). By specifying a profile name or providing access key ID, secret access.

It’s important to note that the credentials specified explicitly in the Sceptre configuration files or directly in the AWS CLI commands take precedence over other methods for credential resolution.
By following this order, Sceptre ensures that it can automatically access the correct AWS credentials based on the environment or explicit settings, making it easier for users to manage and deploy AWS resources with different sets of credentials in various scenarios.

Creating the directory structure for a new Sceptre project

Sceptre provides a convenient command, sceptre new, which can be used to create the directory structure and initial configuration files for a new Sceptre project. This command streamlines the setup process and ensures that you start with a well-organized project layout. To create a new Sceptre project using the sceptre new command, follow these steps:

Open your terminal or command prompt.
Navigate to the location where you want to create the new Sceptre project directory.
Run the following command:

sceptre new project first-sceptre-project

The above command will result in the following directory structure:

  my_sceptre_projects
  ├──first-sceptre-project
     ├── config
     │   └── config.yaml
     └── templates

In the Sceptre project, you will find the configuration directory where you can store the configurations for your Stacks, and the templates directory is designated for holding your CloudFormation templates

Using Docker for Sceptre

For those who prefer containerization, running Sceptre in a Docker container is an excellent option. First, ensure you have Docker installed on your system.

To pull the official Sceptre Docker image from Docker Hub, use the following command:

`docker pull cloudreach/sceptre`

After downloading the image, either create a new directory for your Sceptre project or navigate to the existing project directory in your terminal. Let’s navigate to the Sceptre project directory named “first-sceptre-project” that was created in a previous step.

cd first-sceptre-project

Now, you can run Sceptre commands inside the Docker container using the following syntax:

`docker run -v "$(pwd)":/project -w /project cloudreach/sceptre [sceptre-command]`

For example, to check the sceptre version, run:

`docker run -v "$(pwd)":/project -w /project cloudreach/sceptre --version`

This command mounts your current project directory inside the container and sets it as the working directory for Sceptre commands.

If a custom ENTRYPOINT is desired, you can modify the Docker command accordingly:

docker run --entrypoint='' -it --rm -v "$(pwd)":/project -w /project cloudreach/sceptre sh

By running the command above, you will enter the Docker container’s shell, allowing you to execute Sceptre commands, which is particularly useful for development purposes.

Instead of pulling down the Sceptre Docker image from Dockerhub, an alternative approach is to build a custom Docker image.

Create a Dockerfile with the following content:

 FROM python:3.9

 RUN pip install sceptre

 ENTRYPOINT ["sceptre"]

Next, build the Docker image by running:

docker build -t sceptre-image .

With our custom-built Docker image ready, we can now utilize it to execute Sceptre commands in our project directory, as previously explained in the steps.

Conclusion

Sceptre provides a robust and flexible solution for managing cloud infrastructure as code. In this article, we explored various methods of installing and configuring Sceptre, including running it inside a Docker container. The pip installation method is straightforward and is ideal for those who prefer working directly on their system. On the other hand, Docker offers a containerized approach, providing isolation and consistency for Sceptre commands.

Regardless of the method you choose, Sceptre simplifies cloud infrastructure management, enabling you to define, deploy, and manage AWS resources efficiently. Incorporate Sceptre into your workflow to unleash the true potential of cloud infrastructure as code and take your cloud management practices to new heights.

Remember to consult the official Sceptre documentation for more advanced features and customization options. Happy coding!

Sceptre: The Powerful Infrastructure as Code Tool for AWS

🇦🇺 ☁️ Rumesh Silva ☁️ 🇦🇺 — Wed, 26 Jul 2023 03:43:52 +0000

Sceptre: The Powerful Infrastructure as Code Tool for AWS

In today’s fast-paced cloud computing environment, Infrastructure as Code (IAC) has emerged as a fundamental practice for efficiently managing and provisioning cloud resources. By defining infrastructure in a code-like format, IAC tools enable developers and DevOps teams to automate the process of creating, modifying, and managing cloud resources. Among the numerous IAC tools available, Sceptre stands out as a powerful and flexible choice specifically designed for AWS (Amazon Web Services) environments. In this article, we will explore the features and benefits of Sceptre as an IAC tool for AWS.

What is Sceptre?

Sceptre is an open-source IAC tool developed by Cloudreach that provides a simple yet robust way to define cloud infrastructure as code for AWS. Built on top of AWS CloudFormation, Sceptre leverages the power and versatility of CloudFormation templates while offering a higher level of abstraction and additional functionality to make cloud resource management more intuitive and efficient.

Reasons for choosing Sceptre over CloudFormation, AWS CLI, and Boto3 include its ability to streamline stack deployment, simplify the management of CloudFormation templates, support chaining stack outputs to parameters, offer seamless handling of role assumption and multi-account scenarios, and provide a comprehensive, user-friendly tool to manage AWS infrastructure deployments.

Key Features of Sceptre

YAML Configuration: Sceptre uses YAML configuration files to define cloud resources, allowing for human-readable and version-controllable code. This ensures that infrastructure changes are transparent and easy to track.
Stacks and Templates: In Sceptre, you define infrastructure in reusable templates and organize them into stacks. Stacks are logical groupings of resources, making it convenient to manage complex infrastructures with ease.
Cross-Stack References: Sceptre enables easy referencing of resources across different stacks. This allows for better modularization and reduces duplication of code, promoting best practices in IAC development.
Powerful Hooks: Sceptre allows you to execute pre and post-stack creation/update hooks, giving you the ability to customize deployments further. This feature is particularly useful for integration with third-party tools and scripts.
Parameter Handling: The tool offers advanced parameter handling, making it effortless to pass dynamic values to CloudFormation templates and modify stack configurations as needed.
Multiple Environments: With Sceptre, you can easily manage multiple environments (e.g., development, staging, production) by defining separate environment-specific configurations, providing better isolation and control.

Core Elements of Sceptre

The core elements of Sceptre can be outlined as follows, with Templates and Config being two essential components that must be defined

Stacks: Stacks are the core building blocks in Sceptre. A stack represents a collection of AWS resources that are provisioned and managed together. Stacks are defined using templates written in JSON or YAML format.
Templates: Templates define the desired state of your infrastructure. They specify the AWS resources you want to create, configure, and manage. Sceptre supports multiple template formats such as JSON, YAML, Jinja2, and Python DSLs like Troposphere.
Config: Config files provide a way to configure and parameterize your stacks and templates. Config files allow you to define reusable variables, manage input parameters, and specify stack dependencies.
Hooks: Hooks are scripts or commands that can be executed at specific stages during the stack lifecycle. They allow you to perform custom actions, such as pre or post-stack creation/update tasks.
Resolvers: Resolvers are used to resolve variables or references in the templates and config files. They provide flexibility and dynamic behavior to your infrastructure definitions. Sceptre supports various resolvers, including environment variables, S3 bucket contents, and more.
Overrides: Overrides allow you to modify specific parameters or properties of your stack or template during the creation or update process. They provide a way to customize the behavior of your infrastructure without modifying the original templates.

Benefits of Sceptre for AWS IAC

Simplified Cloud Resource Management: Sceptre abstracts away some of the complexities of AWS CloudFormation, making it easier for developers and DevOps teams to create and manage resources in AWS without diving deep into CloudFormation syntax.
Reusability and Modularity: The ability to define reusable templates and cross-stack references promotes code reusability, leading to more maintainable and scalable IAC codebases.
Flexibility and Customization: Hooks and parameter handling offer a high degree of flexibility and customization, enabling seamless integration with existing tools and the ability to tailor stacks to specific use cases.
Version Control and Collaboration: Storing infrastructure definitions as code in YAML format facilitates version control and fosters smoother collaboration among team members, enhancing overall development practices.
AWS Best Practices: Sceptre encourages adherence to AWS best practices by providing a structured and organized way to manage cloud resources, promoting consistency and reducing potential misconfigurations.

Conclusion

As AWS continues to be a leading cloud service provider, the need for efficient IAC tools becomes increasingly vital for seamless infrastructure management. Sceptre fills this role by offering a user-friendly and powerful solution for defining AWS infrastructure as code. Its simplicity, flexibility, and compatibility with AWS CloudFormation make it a standout choice for teams seeking to harness the benefits of IAC in their AWS environments.

If you haven’t explored Sceptre yet, now is the time to do so. Embrace the power of Infrastructure as Code with Sceptre and unlock new possibilities for automating, scaling, and optimizing your AWS infrastructure. Happy coding and best of luck on your cloud journey!