Forem: Daniel Kim

Using cfn-lint to validate your CloudFormation template

Daniel Kim — Sat, 30 May 2020 11:22:05 +0000

If you've used CloudFormation before you'll already know about the validate-template command on AWS CLI. If you've used this command before, you might also know that this validator only checks the syntax of your template. I can think of countless cases where my deployments failed due to a simple mistake that wasn't caught by this validator. This post will use a simple CloudFormation template to explain how cfn-lint can help you identify various errors in your stack before deploying it.

Sections

Example Template Setup
cfn-lint Usage

Example Template Setup

AWSTemplateFormatVersion: "2010-09-09"
Description: Example template
Parameters:
  Environment:
    Type: String
    Default: 'dev'
Conditions:
  IsDev: !Equals [ !Ref Environment, 'dev' ]
Resources:
  Function:
    Type: AWS::Lambda::Function
    Condition: IsDev
    Properties:
      Role: !Ref FunctionRole # Role property requires an ARN, not the resource ID
      Handler: index.handler
      Code:
        S3Bucket: my-bucket
        S3Key: function.zip
      MemorySize: 3000 # Invalid memory size
      Runtime: nodejs12.x
      Timeout: 5
      TracingConfig:
        Mode: !If [IsTracingActive, 'true', 'false'] # Undefined condition IsTracingActive
  FunctionRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - lambda.amazonaws.com
            Action:
              - 'sts:AssumeRole'
      Policies:
        - PolicyName: all
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action: '*'
                Resource: '*'
Outputs:
  FunctionArn: # Export without matching condition to resource
    Description: Lambda ARN
    Value: !GetAtt Function.Arn
    Export:
      Name: !Sub ${Environment}-lambda-arn

This is the template that we'll use to test cfn-lint. It sets up a simple Lambda function with an IAM role and the lambda ARN is exported as an output. Before we start, let's highlight some of the issues with this template:

Role property on the lambda is not an ARN. Running a Ref on an IAM role will return the logical ID of the role.
MemorySize of 3000 is invalid. Lambda memory size can range from 128 MB to 3008 MB in 64 MB increments.
Condition IsTracingActive is undefined.
LambdaFunction is only deployed when the condition IsDev is true, but it's export does not have the condition defined.

Here's the output of the built-in validate-template command for this stack:

{
    "CapabilitiesReason": "The following resource(s) require capabilities: [AWS::IAM::Role]", 
    "Description": "Example template", 
    "Parameters": [
        {
            "DefaultValue": "dev", 
            "NoEcho": false, 
            "ParameterKey": "Environment"
        }
    ], 
    "Capabilities": [
        "CAPABILITY_IAM"
    ]
}

As you can see, the validator does not identify any issues with this stack. Deploying this stack however will result in a failure. Let's see what cfn-lint can do for us in the next section.

cfn-lint Usage

cfn-lint is a command line tool which examines your CloudFormation template and returns various suggestions. Let's run this tool and see what it has to say about our example stack.

> cfn-lint template.yml

E3008 Property "Role" has no valid Refs to Resources at Resources/Function/Properties/Role/Ref
template.yml:14:7

E2530 You must specify a value that is greater than or equal to 128, and it must be a multiple of 64. You cannot specify a size larger than 3008. Error at Resources/Function/Properties/MemorySize
template.yml:19:7

E8002 Condition IsTracingActive is not defined.
template.yml:23:9

W1001 GetAtt to resource "Function" that may not be available when condition "IsDev" is False at Outputs/FunctionArn/Value/Fn::GetAtt
template.yml:47:5

As you can see, cfn-lint output covers all 4 issues that we previously highlighted in the form of warnings and errors. Errors indicate issues that will prevent your CloudFormation stack from deploying, and warnings indicate potential issues in your stack that you might want to think about.

It's worth noting that not all warnings are necessarily highlighting an issue that you must fix. Everything depends on the context. If you wish to ignore warnings from the output, you can use the ignore_checks argument:

> cfn-lint template.yml -i W

cfn-lint also comes with extensive configuration support which allows you to customise your linting rules.

For a complete set of commands and installation guides, please refer to the github repository.

Conclusion

We looked at how cfn-lint can be used as an additional validator for our CloudFormation templates. It provides a way to check the semantics of our templates and helps us catch mistakes before making a deployment. Some of the CI pipelines that I work on uses cfn-lint to decide whether the deployment step should execute. I hope this tool can help you iterate quicker and make your CloudFormation development experience a bit more enjoyable.

How to secure your S3 bucket from users with s3:* access

Daniel Kim — Fri, 24 Apr 2020 11:14:28 +0000

I recently worked on adding an extra layer of access control for some of the S3 buckets at work. This required me to allow only a specific group of employees to have direct access to the buckets. The problem is that a lot of them can assume an IAM role which gives them s3:* capabilities. In this post I will use an example of two users and IAM roles with read access to all buckets to show how we can restrict access for one user.

Who is this post for?

This post is for you if you:

Want to restrict access to certain S3 buckets even if users have read access to all buckets through their IAM role.

Sections

Setup
Bucket Policy
Testing

Setup

In this section we'll set up some mock data which will help us test our bucket policy. This includes creating new users, policies, groups, and roles. You may skip this section and move on to Bucket Policy if you already have this data.

Let's start off by creating an S3 bucket. I used the default settings since we don't need the bucket to be public for this example.

Navigate to IAM and create two roles with AmazonS3ReadOnlyAccess policy. This policy will give the roles read only access to all buckets.

We will now create policies to allow our users to assume these roles. Go to Policies menu and create two policies. For both policies, select STS as the service and AssumeRole as the action. For resources, click on Add ARN.

Type in each of the two role names for each policy as well as your AWS account ID.

Navigate to Groups to create two groups. Attach one of the policies to each of these groups.

Lastly, navigate to Users. As you may have guessed, we'll be creating two users. Make sure that AWS Management Console access is ticked.

Add each of the users to separate groups that you previously created.

If you've been following the post, you'll now have:

Two roles with AmazonS3ReadOnlyAccess policy
Two policies with sts:AssumeRole access to one of the two roles
Two groups with one of the two policies above
Two users in one of the two groups above

We now have all the data to write our S3 bucket policy.

Bucket Policy

In this section, we'll add a bucket policy which allows specific IAM roles to have read access.

Before we start with the bucket policy, we need to know the ID of the IAM role that should have access to our S3 bucket. We'll use AWS CLI to achieve this. If you don't have this set up, I recommend reading this article from AWS.

Once you have AWS CLI set up, run the following command on your terminal:

> aws iam get-role --role-name YOUR_ROLE_NAME
{
    "Role": {
        "AssumeRolePolicyDocument": {
            "Version": "2012-10-17", 
            "Statement": [
                {
                    "Action": "sts:AssumeRole", 
                    "Effect": "Allow", 
                    "Condition": {}, 
                    "Principal": {
                        "AWS": "arn:aws:iam::12345678:root"
                    }
                }
            ]
        }, 
        "MaxSessionDuration": 3600, 
        "RoleId": "AROACHA7SAS77861CKA", 
        "CreateDate": "2020-04-24T08:23:30Z", 
        "RoleName": "access", 
        "Path": "/", 
        "Arn": "arn:aws:iam::12345678:role/access"
    }
}

Take note of the RoleId. We will need to use this information in our bucket policy.

Bucket policy is a JSON based IAM policy which can define entities that can/cannot access the content of your bucket. The bucket policy that we will add today looks like this:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Deny",
            "Principal": "*",
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::${BUCKET_NAME}",
                "arn:aws:s3:::${BUCKET_NAME}/*"
            ],
            "Condition": {
                "StringNotLike": {
                    "aws:userId": "${ROLE_ID}:*"
                }
            }
        }
    ]
}

This policy denies all requests on the bucket if the requester is not using a specific IAM role. You'll need to manually fill out the BUCKET_NAME and ROLE_ID. The :* on the condition allows us to filter for any sessions that assume this role. This is because the userIdincludes random session identifiers when assuming a role.

Head over to your S3 bucket and navigate to Bucket Policy under Permissions. Add your bucket policy and click Save.

If you've come this far, your bucket should now be accessible by only the IAM role that you've specified. Next section will show an example of how you can test this.

Testing

Head over to the AWS Management Console Login page. Select IAM user and enter your account number. Enter the credentials for the user that can assume the IAM role which should not have access to your bucket.

Open the menu on the top right corner and click Switch Role.

Enter your account number again, and the name (not the ARN) of the IAM role that this user should be able to assume. Click Switch Role. Once you've successfully assumed the role, you'll see an indicator on the top right corner.

Navigate to S3 and confirm that the user does not have access to your S3 bucket. This can be seen from the Error message under Region. You can also click on the bucket to confirm that the user cannot see the content of the bucket. Repeat this process for the user that should have access to your bucket.

Conclusion

We've looked at how we can use bucket policies to prevent people from accessing your S3 bucket even if they have access to read from all S3 buckets. We use an explicit deny policy which is scalable, since we don't need to update the policy as new IAM roles are added to the AWS account.

Using Cognito User Pool as an OpenID Connect Provider

Daniel Kim — Tue, 14 Apr 2020 10:56:49 +0000

Couple of weeks ago I was involved in a hackathon which involved integrating two Cognito user pools together. During my investigation I discovered that Cognito user pool supports OAuth2 for user authentication. This post will outline how you can use Cognito user pool as an OIDC provider as well as how you can connect two user pools together.

Who is this post for?

This post is for you if you:

Have basic understanding of AWS Cognito user pools
Want to know how Cognito user pools can be used as a OIDC provider
How to associate two user pools together through federated identity providers

Sections

What is OpenID Connect?
User Pool Setup
Configuring Federated Identity Provider
Testing

What is OpenID Connect?

OpenID is a protocol which provides a mechanism to authenticate users through a third party. This means that users can login to your application with existing accounts on other platforms. When a user completes the OIDC flow, your application can gain access to some of their information from the third party in the form of an id_token which is a JWT token.

User Pool Setup

In this section we'll create two Cognito user pools and configure them so they can integrate together. oidc-pool-1 will be the master user pool which uses oidc-pool-2 as a federated identity provider. We'll create a user in oidc-pool-2 and keep oidc-pool-1 empty.

Search for Cognito on the AWS console and click on Manage User Pools. Click Create a user pool on the top right corner.

After entering the name of your user pool, you are presented with two options. I'll be clicking Review defaults for the purpose of this demo but you're welcome to step through the settings. This option will let you specify some crucial settings for your user pool such as required attributes for a user. Note that this information cannot be changed once you create your user pool.

Once you've created a user pool, navigate to App clients.

An app client has access to invoke certain endpoints on your user pool while being unauthenticated. Decide on a name and create the app client. We'll leave rest of the settings as is.

Head over to the Domain name page and register a domain for your user pool. Take note of the domain that you enter here.

Repeat the process above and create another user pool. Apply the exact same configuration as the first user pool. Take note of the App client id and App client secret as well as the Pool Id for the second user pool, we will need this when we integrate the two user pools together.

If you followed the post up to this point, you'll now have two Cognito user pools, each with an app client and a domain. If you're satisfied with your setup, navigate to App client settings of your second user pool.

This is where the integration begins. Choose Select all for Enabled Identity Providers and set the Callback URL(s) in the following format:



https://${USER_POOL_DOMAIN}/oauth2/idpresponse

This allows users authenticated with the second user pool to be redirected to the master user pool.

Since I didn't prepare any web app to demo this integration, I'm setting a random value on the Sign out URL(s).

Select Authorization code grant and openid under OAuth 2.0 and click Save changes.

Configuring Federated Identity Provider

Cognito user pools support integration with federated identity providers such as Google and Facebook. We'll use this feature to integrate our user pools together.

Navigate to Identity providers on the first user pool. Select OpenID Connect and you'll see the form below.

Here's how you should fill in the form:



Provider name: Arbitrary name
Client ID: App client id of the second user pool
Client secret: App client secret of the second user pool
Attributes request method: GET
Authorize scope: openid
Issuer: https://cognito-idp.${REGION}.amazonaws.com/${SECOND_USER_POOL_ID}

User pool ID can be found on the General settings menu. Click on Run discovery once you've filled in the form.

Don't be worried if you see this warning.

Open a new tab in your browser and navigate to https://cognito-idp.${REGION}.amazonaws.com/${SECOND_USER_POOL_ID}/.well-known/openid-configuration. You'll see a JSON response with a set of properties. This information can be used to fill in the rest of the form.



Authorization endpoint: https://${SECOND_USER_POOL_DOMAIN}/oauth2/authorize
Token endpoint: https://${SECOND_USER_POOL_DOMAIN}/oauth2/token
Userinfo endpoint: https://${SECOND_USER_POOL_DOMAIN}/oauth2/userInfo
Jwks uri: https://cognito-idp.${REGION}.amazonaws.com/${SECOND_USER_POOL_ID}/.well-known/jwks.json

Click on Create provider Once you've filled in the form completely.

Navigate to App client settings on your master user pool. Select the name of the federated identity provider that you just added (and Cognito User Pool if you want to support direct login through this client). Add suitable URLs for callback and sign out. Check Authorization code grant and openid in OAuth 2.0 and click on Save changes. There is only one step left to integrate the two user pools.

Navigate to Attribute mapping page of the master user pool. Click on the OIDC tab. This form indicates how the attributes of the second user pool maps to the attributes of the master user pool.

When you land on this page for the first time, you'll only see a sub attribute. This is a built-in mapping which acts as a unique identifier of the user. If you're using the default user pool configurations you can add the email related mappings as can be seen above. Otherwise you'll want to map whatever attribute makes sense for your use case. Note that you must add attributes that are marked as required on your master user pool. Click on Save changes once you're happy with the attribute mapping.

This marks the end of this integration! Continue to the next section if you want to find out how you can test it using Cognito Hosted UI.

Testing

To simply the testing process, we'll be using Cognito Hosted UI. This is a built-in set of UI elements that allows users to perform basic auth flows such as sign-in, sign-up and sign-out. In reality, you might be using a custom web app with a library such as AWS Amplify to simplify the authentication process.

Navigate to Users and groups on the second user pool and create a user. For the sake of this post I'll have Mark email as verified ticked. If you're using the same user pool configuration as I am, your user will be in FORCE_CHANGE_PASSWORD state.

Navigate to App client settings on your master user pool and click on Launch Hosted UI.

If you selected both YOUR_OIDC_PROVIDER_NAME and Cognito User Pool on your *App client settings, you'll see the same page as I do. Click on the button below Sign in with your corporate ID.

From this page you can sign in as the user you've created on your second user pool.

Depending on your user pool configurations, you might see this page when you sign in. Simply enter a new password for your user. After this step you may notice that your user in the second user pool is now in a CONFIRMED state. Try logging in again on the Hosted UI.

If you're redirected to google.com (or whatever your redirect URL on the master user pool is) with the code in the query parameter, you've set everything up correctly and the integration is working as expected! Head over to Users and groups on your master user pool to confirm that a new user has been created (it was previously empty). This user should have correct attributes pulled from the second user pool.

Conclusion

We've seen how Cognito user pool can be used as an OIDC provider as well as how we can use this concept to associate two user pools together. If you want to integrate your user pool with another service, you can use the same endpoints provided on this post.

Lastly if you want to integrate this flow with a web app, I again recommend looking into AWS Amplify and how it handles authentication.

Integrating AWS CloudFront with Third Party Domain Provider

Daniel Kim — Tue, 07 Apr 2020 12:05:44 +0000

I recently had to host a website with CloudFront, S3 and a domain from NameCheap. To my surprise, this process was not as simple as I initially thought. There was a lot of conflicting information spread across the web and I had to do some digging in order to get it just right. I'd like to share the exact steps that I followed to integrate CloudFront with a third party domain provider.

Who is this post for?

This post is for you if you:

Have a website hosted on S3 through static web hosting
Have a domain from a non-AWS domain provider such as NameCheap
Want to set up bare domain (non-www) for your website
Want to set up HTTPS redirect on your website
Want to keep your S3 bucket private

Sections

Request Certificate from AWS Certificate Manager
Block public access on S3
Create CloudFront Distribution
Link CloudFront distribution with DNS Provider

Request Certificate from AWS Certificate Manager

A website needs an SSL certificate in order to use HTTPS. The certificate is issued by a Certificate Authority. AWS Certificate Manager (ACM) is a service which, in their own words, handles the complexity of creating and managing public SSL/TLS certificates for your AWS based websites and applications. We will use ACM to generate a certificate that we can later use in our CloudFront distribution.

If you don't have any certificates, this is the screen that you'll see when you navigate to ACM. Before you go any further, please note that you must be in the us-west-1 region. CloudFront only supports ACM certificates from this region. This does not mean that your CloudFront distribution has to be in us-west-1. Click Get Started under Provision Certificates to continue.

Select Request a public certificate and click Request a certificate.

Add your domain names. If you want to set up a bare domain, click on Add another name to this certificate and enter the it on the new text field.

This is how it would look if I wanted to add www.example.com and the bare domain example.com.

Select a validation method. Since this post assumes that you have full control of your DNS configuration, we'll select DNS validation.

Add relevant tags and confirm the request once you've reviewed it.

If you followed the post up to this point, you'll see a screen like this:

ACM provides a Name/Value pair for each domain that you've added. As you may have noticed, they follow a certain format.



Name=_RANDOM_STRING_0.DOMAIN_NAME.
Value=_RANDOM_STRING_1.RANDOM_STRING_2.acm-validations.aws.

All of the domains are in Pending validation status. In order to prove that you own these domain names, you'll need to add these as a CNAME record. Head over to the DNS configuration page of your domain provider. This is called Advanced DNS on Namecheap.

The exact Name/Value that you need to enter on this page varies between each domain provider. During my investigation, I noticed two main variations:

Some DNS providers don't allow a leading underscore.
Some DNS providers append host name to the key automatically.

For the sake of this post, let's say that you have the following Name/Value pair from ACM:



Name=_1234567890.www.example.com.
Value=_0987654321.abcdefg.acm-validations.aws.

You should format your CNAME record in the following way:

If leading underscore is not allowed, simply remove it
If host name is automatically populated, remove .hostname.com. from the Name. Following these rules, the resulting Name could look like: ```

Name=_1234567890.www


![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/idwqu703wux31p792po6.png)
While we're at it, we'll also add a URL Redirect record to support bare domains. Note that the *@* symbol indicates the current domain.


Once you add the CNAME records, you need to wait until ACM validates them. However, you probably want to make sure that you've set them up correctly. You can achieve this with dig.

dig cname FULL_NAME_FROM_ACM


If you've done everything correctly, you'll see the correct Value from ACM on the ANSWER SECTION of the response. Note that it can take a couple of minutes for your records to propagate.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/luw1qci8yg0m41dda13h.png)
When ACM successfully validates your domain name, your certificate will be in *Issued* status.


# Block public access on S3 <a name="chapter-2"></a>
This post assumes that you already have a website hosted on S3 through static web hosting. We'll go over how to make sure that only CloudFront can access your bucket. If you don't need to block public access to the bucket, you can skip this section.


In order to deny anyone other than CloudFront from accessing your bucket, we'll set up an explicit deny on the bucket policy. This will include a random referer header:

{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "PublicReadGetObject",
"Effect": "Deny",
"Principal": "",
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::your-bucket-name/",
"Condition": {
"StringNotEquals": {
"aws:Referer": "1234567898767543321123"
}
}
}
]
}


Generate a random string for your header value and save the bucket policy. Take note of this value, you'll need it when you create your CloudFront distribution



# Create CloudFront Distribution <a name="chapter-3"></a>
Navigate to CloudFront page on the AWS console and click on *Create Distribution*. Since we want to host a website, you'll want to create a Web distribution.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/4wviq6ma79qp1sidhgrz.png)

Most of the fields on this form should be filled out according to your needs, so I'll focus on the key fields for this integration.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/imwfbldh0pufuzt2q8nu.png)

This is where you add your referer header. The value should match the random string in your bucket policy. Note that *aws:* is omitted from the header name.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/6vh58jn4aer73h40lyxn.png)

Select the certificate that you requested. Note that your certificate has to be in the *Issued* status for this to work.

Fill in the rest of the form and create the distribution.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/dlhejwp0v18ipooel0pc.png)

You'll need to wait until your distribution is in the *Deployed* state. This may take some time. Take note of your distribution's domain name, you'll need this soon.

# Link CloudFront distribution with DNS Provider <a name="chapter-4"></a>
This is the last step of the integration. If you've come this far on your first attempt, you're doing much better than I did.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/qei1prd0i7vlzhp6wz5d.png)

Open your DNS configuration page again and add a new CNAME record. This allows your domain name to point to your CloudFront distribution. Note how I'm using *www* as the host. This is because Namecheap automatically appends the host name. If your DNS provider doesn't support this, you'll want to add the entire domain name.

As you've seen before, it can take some time for this record to propagate. You can use the dig tool to validate your configuration.


# Conclusion
And that's it! Your CloudFront + S3 + third party domain provider integration should now work while keeping your S3 bucket from being accessed by anyone else. For something that sounds so simple conceptually, it took me an embarrassing amount of time to get this right. Many of the configurations in this integration can take a long time to propagate, so trial and error can become a real hassle. I hope this post can help people save some time.