Forem: Justin Perkins

All Aboard the AMI Express: Systems Manager Parameter Store

Justin Perkins — Mon, 19 Dec 2022 19:51:25 +0000

We have all been there before, well at least I have, multiple times now in fact. Send off your old faithful CloudFormation template that creates a few EC2 instances and a few seconds later hit with the: “API: ec2:RunInstances Not authorized for images: ‘ami-12345abcdef’ …”. I hope for some of you out there this hasn’t occurred and hopefully after reading this will never occur. For others enjoy some coffee and hear my struggles of keeping my stack AMI values up to date.

In this article, I am going to share with you an automation trick one of my colleagues enlightened me too, which involves a crafty way of always retrieving the latest AMI-ID published by AWS to be used for my deployments. By using AWS Systems Manager Parameter Store, I can use unique identifiers to specifically select my desired AMI-ID.

What is an AMI?

An Amazon Image ID or AMI is a template that encompasses the required software to launch an EC2 instance. From theses AMI’s you can launch a instance, and what is created is an instance of that AMI, so a running copy. When selecting an AMI, you can select either an:

AWS Provided AMI — supported and maintained by AWS.
Paid AMI — Purchase a developer provided AMI on AWS MarketPlace.
Community AMI — AMI’s shared with the community or public.
Custom AMI — Using existing AMI’s customize it further with required application and security software. Creating a Golden-Image to launch instances from. You can also create and sell AMI’s.

Any time you create an EC2 instance whether it be by console, Command Line Interface (CLI), or infrastructure as code (IaC) you must specify an AMI at launch.

Some common characteristics to look for when selecting an AMI:

AWS Region
Operating System (OS)
Architecture (32 vs. 64 bit)

Note: For Linux AMI’s you can also define a virtualization type depending on what the needs are. Linux offers hardware virtual machines (HVM) as well as paravirtual (PV). See here for more information regarding Linux virtualization types.

What is Infrastructure as code?

Infrastructure as code or IaC is a process of provisioning and managing cloud resources through code. To use IaC, a configuration (blueprint) template file needs to be defined that contains instructions for the exact resources and specifications to build and deploy projects. With the building of these IaC templates, many benefits can be achieved such as improved consistency, version control, and faster time to deployment. Some popular IaC services are:

The Problem at hand:

I built and continue to build a lot of IaC templates that create various EC2 instances that range in OS flavors and releases. A lot of these templates are used in workshops I have created to highlight different security concepts and workflows. A work colleague of mine was using one of the previous workshops I built and if you are familiar with the infamous Murphy’s law [anything that can go wrong, will go wrong] when it comes to live demos then as you can guess he ran into an error stating “the image id ‘ami-12345abcdef’ does not exist”. This error happens periodically for a few reasons. AWS performs OS updates to their AMI’s they offer to customers. Each AMI is available to a particular region as well. So when these updates occur a new AMI ID is allocated while the older one is set to be deprecated.

Now, this is something that has occurred to me before, so I already knew the solution was to manually go into AWS EC2 and just grab the latest AMI-ID for that specific OS and update the CloudFormation stack with the value and I am good to go for a brief while. However, I was building multiple different workshops each with its own needs. This became a weekly event for me to ensure all my templates had updated and accurate AMI’s, this grew to become a pain as the templates scaled.

In my previous iterations, I chose to leverage a technique that involved mappings. In CloudFormation, this allowed me to set values based on a specific AWS region it was to be deployed. In my case, I could define a region(s) with AMI-ID values for each OS that was needed in my case. AWS has 27 regions, I covered just 4 using this approach. In addition, each time I updated the AMI-ID values for each respective OS in each respective region I was also subjected to potential human error with copy and paste. A lot of different things could go wrong and worst of all, it was a monotonous process that went directly against the DRY method or “Don’t Repeat Yourself” method.

Solution:

The very same work colleague that had to navigate the template error also later enlightened me on a better approach when deploying different OS types and as well as easier management scaling availability for a global audience. He suggested I update my template and abandon the mappings approach in lieu for Systems Manager Parameter Store.

With SSM Parameter Store, it turns out that it hides a gold mine of strings that can be queried to return the latest AMI image ids for different OS and releases. All that was needed on my end was to locate the needed string values. I navigated to SSM Parameter Store, selected the option for “Public Parameters” and selected the service for which I am looking for values to leverage.

Once I acquired my OS-specific parameter string values, all that is left is to update my IaC template to reflect this new approach. In the example below, I created a parameter for each OS that was needed, in this case an Ubuntu 18 AMI and an Amazon Linux AMI. Defined in the example is a default value for each respective parameter. For each OS, I applied the value acquired from my Systems Manager Parameter Store search.

Description: A template to deploy a Linux & Ubuntuinstance.
#High lights SSM parameter store public AMI-ID strings.
Parameters: 
  #CFT Parameter created with default string for SSM AMI string.
  LatestUbuntuAmiId: 
    Type: 'AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>'
    Default: '/aws/service/canonical/ubuntu/server/18.04/stable/20220810/amd64/hvm/ebs-gp2/ami-id'
    Description: DO NOT CHANGE THIS VALUE
Resources:
  #Amazon Linux Resource created with resolve string for SSM AMI string.
  AmazonLinux: 
    Type: AWS::EC2::Instance
    Properties: 
      ImageId: resolve:ssm:/aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2
      InstanceType: t3.micro
      Tags:
      - Key: Name
        Value: AmazonLinux
      - Key: Project
        Value: SSM
  #Ubuntu Resource via CFT input parameter to link for SSM AMI string.
  UbuntuLinux: 
    Type: AWS::EC2::Instance
    Properties: 
      ImageId: !Ref LatestUbuntuAmiId
      InstanceType: t3.micro
      Tags:
      - Key: Name
        Value: Ubuntu
      - Key: Project
        Value: SSM

https://gist.github.com/JustinDPerkins/09001a21f974432ecb57d7c29daa7a25

Conclusion:

Some struggles do come to an end. I have yet to see the issue recur now for a few months and it is now a method I employ on all new templates I create that require EC2 instances. I encourage anyone who is currently developing or planning on developing to employ the AMI-ID retrieval using SSM parameter store. Using this method will help prevent you from dealing with failed deployments and that irritating AMI not found error. Last, I want to thank my colleague Raphael Bottino for shedding light on this awesome trick and saving me countless future headaches, so make sure to give him a follow as well.

Image Credit:
Unsplash- Ankush Minda - https://miro.medium.com/max/640/0*Yw785vmT_KD5Yx1S

Fall of the Death Star — Why Scanning Infrastructure as Code Matters

Justin Perkins — Fri, 04 Nov 2022 19:20:00 +0000

https://pixabay.com/illustrations/ship-space-technology-universe-3857479/

The Empire’s greatest and most powerful weapon was the Death Star aka the DS-1 Orbital Battle Station. It was the size of a small moon and struck terror across the galaxy as it was a planet destroyer thanks to the large Kyber-crystal powered super laser mounted on it.

This article will be discussing the importance of implementing security early in the continuous integration & continuous delivery pipelines (CI/CD) to mitigate cloud resource misconfigurations from being introduced. The story of the destruction of the Death Star will be used to aid in depicting poor cloud security practices used by the Empire. By implementing scanning of infrastructure, the Empire could have easily defeated the Rebel Alliance.

Thank goodness in this case the Empire had terrible security practices. Spoiler Alert!

Designing the Death Star

Now, this terrifying monstrosity of a battle station started just as any cloud project would. An idea or need was formed. A team was then delegated to turn this vision into a tangible design to be built, deployed, and, unfortunately for Alderaan, used.

Galen Erso was an engineer tasked to design key resources for use on the Death Star. As the Star Wars lore goes the Death Star had a misconfiguration by design. In the build blueprints, a fatal weakness lies within the Death Star in the form of a thermal exhaust port leading directly to the reactor. This critical misconfiguration was not clearly visible to the Empire.

Think of the Death Star architectural blueprint as an Infrastructure as Code (IaC) template using tools such as AWS (Amazon Web Services) CloudFormation or Terraform to build the Death Star’s cloud project.

The Death Star will represent an AWS EC2 server.
The Reactor will represent a business-critical application running.
The Thermal Exhaust Ports will be the attached security groups.

What is Infrastructure as Code (IaC)?

For those who are new or unfamiliar, IaC is a process of provisioning and managing cloud resources through code. To use IaC, a configuration (blueprint) template file needs to be defined that contains instructions for the exact resources and specifications to build and deploy projects. With the building of these IaC templates, many benefits can be achieved such as improved consistency, version control, and faster time to deployment.

What are misconfigurations?

When using a Cloud Service Provider (CSP) such as AWS, tons of different services are offered to select from to create infrastructure that meets a need. These services are each unique and create resources that are configured by the customer. Imagine creating an S3 bucket to store some private information. However, when creating that bucket it is accidentally set to public. This is a misconfiguration. It leaves the resource exposed to risk.

Thermal Exhaust Ports and the Reactor.

As mentioned earlier the Death Star had flaws by design. The example below is an AWS CloudFormation template. In the template code defined is an EC2 server named the “Death Star” with a security group attached called “Thermal Exhaust Ports”.

Example CloudFormation template depicting a potential misconfiguration involving an EC2 resource.

From a functionality perspective, this template will deploy successfully, so the Empire decides to proceed to build and deployed this monstrosity. However, this lack of security and visibility in the Empire’s CI/CD pipeline would later come back to haunt them when Luke Skywalker exploits this misconfiguration by firing two proton torpedoes into the exhaust ports (Security Groups) that lead directly to the reactor (Running Application). Thus, destroying the once terrifying Death Star.

What is wrong exactly?

The Empire had not yet instilled any security features throughout their CI/CD pipeline to prevent such resource misconfigurations from reaching deployment. In the example above the following misconfigurations went unnoticed:

Security group ThermalExhaustPorts allows unrestricted access to uncommon ports

When building it is important to check the EC2 security groups for inbound rules that allow unrestricted access (i.e. 0.0.0.0/0 or ::/0) to any uncommon TCP and UDP ports. In addition, restrict access to only those IP addresses that are required to implement the principle of least privilege and reduce the possibility of a breach.

Security group ThermalExhaustPorts opens range of ports

Ensuring that security groups do not have a wide range of ports open for inbound traffic in EC2 instances can be better protected against attacks such as denial-of-service (DoS) attacks or brute-force attacks. It is recommended to open only specific ports within your security groups, based on your application’s requirements.

The misconfigured Thermal Exhaust Ports (Security Groups) allowed Luke Skywalker to fire proton torpedos (exploit) at the Death Star’s reactor (Critical application) rendering the Death Star useless.

How could this have been prevented?

The Empire could have leveraged a Cloud Security Posture management solution to span their entire development pipeline in three stages. Code, Build, and Deployment.

Example architectural diagram implementing Cloud Security Posture Management solution throughout the CI/CD pipeline

1. Code — Integrated Development Environment (IDE)

Most developers code on their local machine to start. By integrating a security plugin into the developer’s IDE, immediate feedback can be gained for the IaC developers to leverage. This feedback can be used to fix identified misconfigurations from the scanned template. Scanning IaC templates in the IDE shifts security as far left as possible and helps developers build more secure resources right from the start. Once validated locally the template can be pushed to a code repository for further workflows.

Infrastructure as Code template scan result example of VS Code IDE IaC scan plugin.

Galen Erso was the developer of these misconfigured resources. Although he planted these misconfigurations purposefully in hopes that the Rebels would have any chance of destroying the Death Star. By integrating an IaC security plugin into his IDE. Erso would have been able to without a doubt identify the misconfigurations.

2. Build — Scanning Templates in CI/CD Pipeline

Once a developer has pushed an IaC template to a code repository such as GitHub or AWS Code Commit. A workflow can subsequently be triggered to automatically build and deploy the defined resources. By integrating an IaC template scan provided by a Cloud Security Posture Management tool, CI/CD pipelines can be improved from a security perspective. Leveraging CSPM APIs (Application Programming Interface) can ensure every IaC template submitted in the pipeline is scanned to adhere to the well-architected framework best practices or some other compliance standard. If it does not, then the build is halted. Results from the scan can be sent to notify both developers and operation teams of any potential security misconfigurations.

The Empire should have had a CI/CD pipeline in place. Erso should have completed his Death Star build designs and pushed the template to a repository. The Empire’s build pipeline should have been fitted with a CSPM’s template scanning capabilities to prevent the introduction of misconfigured resources. The pipeline would have been started once the template was pushed. Erso’s design would have been scanned in this stage. The Thermal Exhaust Ports would have been identified as a misconfiguration. The build would be stopped. Lastly, notification would be sent to the Empire’s Developer and Operations teams for further action.

3. Deployment — Cloud Security Posture Management (CSPM)

A CSPM is a security tool that continuously monitors cloud accounts for recent changes and detects existing misconfigurations in multiple cloud service providers. With a CSPM development and operations teams can address security challenges such as current misconfiguration visibility, assist in adhering to compliance, scheduled cloud risk assessments, and audits, as well accelerate incident response teams.

Example CSPM console depicting current cloud infrastructure posture.

The Empire would have had a final chance to gain visibility into the Death Star if they integrated a CSPM tool. The CSPM would have been capable of periodically scanning the Death Star as well to ascertain any misconfigured resources that may already exist. In addition, imagine if the Death Star was initially built without any misconfigurations and deployed. If Galen Erso or anyone made any changes to the Death Star’s configuration state a CSPM with real-time monitoring capabilities would identify that immediately. Then the Operations team would be notified immediately for further action.

Conclusion

Leveraging Infrastructure as Code whether Terraform or CloudFormation has many benefits when it comes to managing cloud resources. However, it is important as a CSP customer to ensure the proper configuration of each resource deployed. If the Empire had integrated security in any of the last two stages mentioned in this article — CI/CD (build) or continuously monitoring deployed resources(deployment), the Rebel Alliance would have been crushed easily. In the examples above the Empire ignored simple security practices that later Luke Skywalker exploited. Do not be like the Empire, integrate security as far left and throughout the development pipeline. May the force be with you.

Follow me on medium

The Who, the What, the Where, and How To Limit the Blast Radius

Justin Perkins — Tue, 01 Nov 2022 20:00:00 +0000

Identity and Access Management or IAM is one of the most important services for cloud customers today. Creating and managing the identities of multiple users and resources and what unique actions each user needs to be able to perform is daunting, but it is also critical in keeping your cloud account and infrastructure secure. You wouldn’t want to build a house on a foundation full of cracks.

In this article, we will elaborate on the various ways you can limit or restrict actions performed on resources or by users, or by roles. We will also discuss some of the different components and features of AWS IAM like permission boundaries and how they can be used in combination to achieve desired authorization while still proactively restricting any unwanted actions or escalation of privilege.

AWS IAM

IAM is a service that aids to control access to resources. IAM is used to control authentication and authorization to your cloud account and resources. So simply put, who is who, and what can they do?

In most cases, IAM is the first security layer that can be implemented when it comes to securing your AWS resources and services. IAM allows us to create and manage identities. An identity can be as simple as a user that needs to interact with AWS services. An identity also can also be allowing one AWS service to interact with another AWS service.

The Who — Identities and Authentication

IAM identities typically revolves around the following components:

IAM Users: Users are entities that are created in AWS to represent a person to allow interaction with AWS. For example, a user named Bob could be created in IAM for an employee named Bob. Users can define a password (please rotate your passwords and configure MFA while your at it) to access AWS. Users also can be granted access keys to perform programmatic calls to AWS. Keep both safe and secure and never share with anyone or post on any repository.

IAM Groups: Groups are collections of IAM users. Using IAM Groups you can bundle multiple users together to share a common set of policies. Our IAM user Bob could be assigned to an IAM group for admins.

IAM Roles: A Role is an identity, much like a user, that can be created to allow interactions with AWS. The difference with a role is that it does not need to be uniquely associated with one person. A role can be used by users, applications, or services like EC2 to assume in order to perform actions on other AWS services such as S3.

By default, both newly created users, groups, and roles have no permissions attached to access resources. A policy will need to be attached in order for the identities to successfully interact with AWS services.

The What — Authorization

IAM Policy: This is where all the actions are defined. These beautiful documents are defined in the easily readable format of JSON. These JSON policies define permissions that either allow or deny certain requests made. It is best practice in AWS to ensure you only grant the required permissions to complete the task or job and nothing more. This is the golden rule known as implementing least privilege.

### IAM Policy attached to user Bob
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "ListObjectsInBucket",
            "Effect": "Allow",
            "Action": ["s3:ListBucket"],
            "Resource": ["arn:aws:s3:::bobsbucket"]
        },
        {
            "Sid": "AllObjectActions",
            "Effect": "Allow",
            "Action": "s3:*Object",
            "Resource": ["arn:aws:s3:::bobsbucket/*"]
        }
    ]
}

Above is an example policy that will be attached to our user named ‘Bob’. In this policy we have two defined action statements. The first is to “Allow” our principal “Bob” to perform an “S3:ListBucket” api call to a specific bucket defined as ‘bobsbucket’. The second part “Allows” to perform any object based call (S3:Object) to the same specific S3 resource “bobsbucket” for any object. The addition of the asterisks () to a policy allows all. Try to limit the inclusion of these when creating your policies.

IAM policies come in two different flavors:

Managed Policies — These can either be AWS managed, meaning AWS creates and manages the policy. Alternatively, you can create a customer-managed policy in which you define and manage the policy, allowing for more precise control. I came across this fantastic website where one can see all AWS managed policies here.

Inline Policies — These are policies attached directly to a single identity, such as a user, role, or group. When the user or role is deleted, the inline policy is also deleted.

An IAM entity with an attached IAM policy will be able to perform only the actions defined in its policy. So, take the policy example above and we will assign it to our user Bob. Bob makes a request to list objects in his S3 bucket. Bob’s permissions will be evaluated and the request will succeed in this case due to no higher priority policy being attached either at the resource or account-level, or a permission boundary in place (these are discussed later).

How are policies evaluated?

Determining whether a request is allowed or denied within an account, for a user, a role or group involves understanding how decision making is accomplished when different types of policies are being used combination. AWS created the flowing policy evaluation flowchart that details how decisions are made depending on the combinations of policy types in play.

The Where — Resource Authorization

Resource-based Policy: Resource-based policies are unique permission policies that are attached directly to a service resource such as S3. The powerful thing about these policies is that you can specify who or what has access to that specific resource and what exact actions can be performed on it. My recommendation is if the AWS service you are leveraging allows a resource-based policy to be attached, apply one.

Using the same user Bob and policy as before we can submit the same request to S3. The code snippet below has been applied to our S3 resource. It allows our ‘principal’ user ‘Bob’ the ability to perform any action on a S3 resource named ‘bobsbucket’ and all its objects. In this case, the resource based policy will be evaluated first as it has the highest priority in this example.

### Resource-Based Policy attached to S3 bucket
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::123456789012:user/bob"
            },
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::bobsbucket/*",
                "arn:aws:s3:::bobsbucket"
            ]
        }
    ]
}

The identity’s or in our case user “bob’s” policy will then be evaluated if no conflicts occurred in the resource policy. As in the prior example our request will be allowed due to both the resource policy allowing this action and the user policy attached to Bob.

How to limit the blast radius

When creating and managing your IAM it is important to place additional layers of security to gain the most control over an entity’s effective permissions. We want to effectively limit our identity’s capabilities to prevent any misuse of permissions or privilege escalations. Effectively building a cap to extinguish any unplanned actions from occurring.

IAM Permission boundaries: A Permission boundary is a feature for setting the maximum allowed permissions that an IAM User or IAM Role can be granted. By leveraging permissions boundaries, you can enforce only the actions that are granted by BOTH the IAM user/role and its defined permission boundary.

For example, assume that the IAM user named Bob should be allowed to interact only with Amazon S3. To enforce this rule, you can use the following policy to set the permissions boundary for the user Bob. With the policy below any action made to an outside service such as IAM or DynamoDB will be denied.

### IAM Permission Boundary Policy Example
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:*"
            ],
            "Resource": [
                "arn:aws:s3:::bobsbucket/*",
                "arn:aws:s3:::bobsbucket"
            ]
        }
    ]
}

Service-Control Policy (SCP): These are used in tandem with AWS Organizations. It is an organizational-level policy to manage permissions. Like permission boundaries, SCPs offer control over the maximum permissions for all accounts in your organization or specific organizational unit (OU). Users and Roles must still be created and have IAM policies associated with them. So, for example, if a user or role has an IAM policy that grants access to S3 and that action is either not permitted or explicitly denied by the SCP, the user or role will not be allowed to perform that action. SCPs only affect IAM users and roles associated WITH the organization. SCPs do not affect resource-based policies directly. So, even if an S3 bucket policy allows an entity to perform some action; if both the SCP and IAM user policies do not allow that action, it will be denied.

Combining the same user Bob and policy as before, also attaching the same permission boundary as before, we can submit the same request to S3. The code snippet below has been applied to our AWS Account. It denies our account the ability to perform 2 actions on any resource. In this scenario, the SCP will first be evaluated, then the resource-based policy, third is the IAM identity Bob’s attached permissions, last for evaluation is the permission boundary. For our scenario, the request would be successful.

### AWS Organization Service Control Policy 
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Deny",
      "Action": [
        "s3:DeleteObject",
        "s3:PutAccountPublicAccessBlock"      ],
      "Resource": "*"
    }
  ]
}

However, lets take the same scenario and change the request. The user Bob now makes a request to delete an object from S3. Using the AWS policy evaluation flowchart the first policy to be evaluated would be our attached SCP. We have an explicit deny for objects in S3 being deleted. So Bob’s request would be rejected even though the user has permissions to perform any S3 actions and the resource grants access to our user.

Both Permission Boundaries and Service Control Policies can limit permissions but do not directly grant/provide permissions. They can be considered guardrail functionality, think of them as using “bumpers” when bowling. The bumpers create a defined channel in which the bowling ball will be allowed to go toward the pins. These IAM guardrails will prevent your trusted users from performing any erroneous actions as well as reduce the blast radius if an IAM entity compromise were to occur escalation of privilege could not proceed any further.

Conclusion

When creating users, groups, or roles be sure to always implement the practice of least privilege. This will ensure all business operations are still successful while effectively limiting our identity’s access to only what is absolutely required to do their job or task. Further, bolster resource-based security by enforcing policy at the resource level to prevent any actions on a resource from being successfully allowed unless stated. By using advanced features like permission boundaries, you can place a guardrail on the IAM identity to set the maximum permissions allowed for an IAM entity. Last is if you own more than one AWS account leverage AWS Organizations and implement service control policies to centrally manage each individual account. When SCPs are applied in conjunction with the above policy types you gain the most cloud-native control. For any request to be allowed would need to be approved by the SCP, the IAM policy, and the permission boundary. By implementing all the policy types mentioned in this article you will gain the most control.

Follow me on Medium