Forem: Cao Duy Khanh

Static Analysis for CloudFormation

Cao Duy Khanh — Wed, 29 Dec 2021 17:38:27 +0000

To satisfy CloudFormation is well, we should apply static analysis tool for it.
Today, I would like 2 static analysis tools:

cfn-lint: Verify that the template is correct.
cfn-nag: Verify that there is no code that poses a security risk.

Sections

Setup
cfn-lint Usage
cfn_nag Usage

Setup

cfn-lint

pip install cfn-lint

cfn_nag

gem install cfn-nag

setup cloudformation template with name s3.yml

AWSTemplateFormatVersion: '2010-09-09'

Description: S3 Cloudformation Template

Parameters:

  EnvironmentStage: 
    AllowedValues:
      - "Dev"
      - "Stg"
      - "Prod"
    Default: "Dev"
    Description: "Environment Dev, Staging, Production"
    Type: String

Resources:

  WebCmsOAI:
    Type: AWS::CloudFront::CloudFrontOriginAccessIdentity
    Properties:
      CloudFrontOriginAccessIdentityConfig:
        Comment: Origin Access Identity for Web Cms S3 Bucket

  WebCmsBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: !Sub '${AWS::StackName}-web-cms'
      CorsConfiguration:
        CorsRules:
        - AllowedHeaders:
          - '*'
          AllowedMethods:
          - GET
          - PUT
          AllowedOrigins:
          - '*'
          MaxAge: 3000
      VersioningConfiguration:
        Status: Enabled
      WebsiteConfiguration:
        ErrorDocument: index.html
        IndexDocument: index.html

  WebCmsBucketPolicy:
    Type: AWS::S3::BucketPolicy
    DependsOn: WebCmsBucket
    Properties:
      Bucket: !Ref WebCmsBucket
      PolicyDocument:
        Statement:
          - Action:
              - 's3:GetObject'
            Effect: Allow
            Resource: !Sub 'arn:aws:s3:::${AWS::StackName}-web-cms/*'
            Principal: 
              CanonicalUser: !GetAtt WebCmsOAI.S3CanonicalUserId

Outputs:
  WebCmsBucket:
    Value: !Ref WebCmsBucket
    Export:
      Name: !Sub '${AWS::StackName}-WebCmsBucket'

  WebCmsBucketEndpoint:
    Value: !Join
      - ''
      - - !GetAtt
           - WebCmsBucket
           - RegionalDomainName
    Export:
      Name: !Sub '${AWS::StackName}-WebCmsBucketEndpoint'

  WebCmsOAI:
    Value: !Ref WebCmsOAI
    Export:
      Name: !Sub ${AWS::StackName}-WebCmsOAI

cfn-lint Usage

> cfn-lint s3.yml
W2001 Parameter EnvironmentStage not used.
s3.yml:7:3

W3005 Obsolete DependsOn on resource (WebCmsBucket), dependency already enforced by a "Ref" at Resources/PublicCMSWebsitePolicy/Properties/Bucket/Ref
s3.yml:46:5

As you can see, cfn-lint output covers all 2 issues that we previously highlighted in the form of warnings and errors. Errors indicate issues that will prevent your CloudFormation stack from deploying, and warnings indicate potential issues in your stack that you might want to think about.

cfn-lint also comes with extensive configuration support which allows you to customise your linting rules.

WebCmsBucketPolicy:
    Type: AWS::S3::BucketPolicy
    DependsOn: WebCmsBucket
    Properties:
      Bucket: !Ref WebCmsBucket
      PolicyDocument:
        Statement:
          - Action:
              - 's3:GetObject'
            Effect: Allow
            Resource: !Sub 'arn:aws:s3:::${AWS::StackName}-web-cms/*'
            Principal: 
              CanonicalUser: !GetAtt WebCmsOAI.S3CanonicalUserId
    Metadata:
      cfn-lint:
        config:
          ignore_checks:
            - W3005

cfn_nag Usage

> cfn_nag_scan -i s3.yml
------------------------------------------------------------
s3.yml
------------------------------------------------------------------------------------------------------------------------
| WARN W35
|
| Resource: ["WebCmsBucket"]
| Line Numbers: [25]
|
| S3 Bucket should have access logging configured
------------------------------------------------------------
| WARN W41
|
| Resource: ["WebCmsBucket"]
| Line Numbers: [25]
|
| S3 Bucket should have encryption option set

Failures count: 0
Warnings count: 2

As you can see, cfn_nag output covers all 2 issues that we previously highlighted in the form of warnings and failures. These issues that will prevent your CloudFormation stack from security risks

cfn_nag also comes with extensive configuration support which allows you to customise your rules.

WebCmsBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: !Sub '${AWS::StackName}-web-cms'
      CorsConfiguration:
        CorsRules:
        - AllowedHeaders:
          - '*'
          AllowedMethods:
          - GET
          - PUT
          AllowedOrigins:
          - '*'
          MaxAge: 3000
      VersioningConfiguration:
        Status: Enabled
      WebsiteConfiguration:
        ErrorDocument: index.html
        IndexDocument: index.html
    Metadata:
      cfn_nag:
        rules_to_suppress:
          - id: W35
            reason: "Test ignore rule S3 Bucket should have access logging configured"
          - id: W41
            reason: "Test ignore rule S3 Bucket should have encryption option set"

Nginx + Private S3

Cao Duy Khanh — Sat, 07 Aug 2021 17:12:45 +0000

How can I view photos on private S3. Today I will show how we can do it with Nginx on AWS. We first need to set up something like below:

VPC(10.0.0.0/16) with a public subnet and a private subnet
VPC Endpoint for s3
A static web on ECS Fargate with Nginx
Private S3 Bucket which store images of application

We will have a static web like below with an image loaded from the internet.

I have an image which store on S3. So I want to own static web will view that image.

Now I will go into setting specific sections.

(1) First we will need to update S3's bucket policy to only allow viewing of images from the vpc endpoint.



{
    "Version": "2008-10-17",
    "Id": "PolicyForCloudFrontPrivateContent",
    "Statement": [
        {
            "Sid": "Access-to-specific-VPCE-only",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::demo-static-s3/*",
            "Condition": {
                "StringEquals": {
                    "aws:sourceVpce": "vpce-0d92e50f230bc8070"
                }
            }
        }
    ]
}

(2) Next we will configure Nginx to be able to view images from S3 as shown below.



location ~^/image/(.+)$ {
        resolver 10.0.0.2;
        proxy_pass http://s3-ap-southeast-1.amazonaws.com/demo-static-s3/image/$1; 
}

resolver: IP address of the DNS server in your AWS VPC

I have VPC with CIDR range is 10.0.0.0/16. So IP address of the DNS server is 10.0.0.2 => Resolver is 10.0.0.2

10.0.0.0: Network address.
10.0.0.1: Reserved by AWS for the VPC router.
10.0.0.2: Reserved by AWS. The IP address of the DNS server is the base of the VPC network range plus two. For VPCs with multiple CIDR blocks, the IP address of the DNS server is located in the primary CIDR. We also reserve the base of each subnet range plus two for all CIDR blocks in the VPC. For more information, see Amazon DNS server.
10.0.0.3: Reserved by AWS for future use.
10.0.0.255: Network broadcast address. We do not support broadcast in a VPC, therefore we reserve this address.

Refer: https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Subnets.html

proxy_pass: s3 link

Format: http://s3-ap-southeast-1.amazonaws.com/[main_bucket]/[sub_bucket]/$1

(3) Next I will change the html to load an image from s3 instead of an online image.



<div class="container">
    <h1>Hey there</h1>
    <img src="https://cleandevs.com/image/668c1d479e27bb8750823655c83a6c9bd90263f9_hq.jpg"/>
  </div>

(4) Finally, I will deploy to ECS Fargate. The result you will see is as below.

It is able to solution to view images from S3 without using CloudFront.

How to config Gitlab CI just checks commit changed files

Cao Duy Khanh — Thu, 05 Aug 2021 13:32:21 +0000

[Background]
I have a Python project with config CI on Gitlab. It has a lint stage which will check code quality of all source code when push event or merge request event is triggered. It will take time to check all source code with merge request. And it will not focus on code quality of merge request. It should check code quality on commit change files.

pylint:
  stage: lint
  before_script:
    - mkdir -p public/badges public/lint
    - echo undefined > public/badges/$CI_JOB_NAME.score
    - pip install pylint-gitlab
  script:
    - pip install -r requirements.txt
    - pylint --rcfile=.pylintrc --exit-zero --output-format=text $(find -type f -name "*.py" ! -path "**/.venv/**") | tee /tmp/pylint.txt
    - sed -n 's/^Your code has been rated at \([-0-9.]*\)\/.*/\1/p' /tmp/pylint.txt > public/badges/$CI_JOB_NAME.score
    - pylint --exit-zero --output-format=pylint_gitlab.GitlabCodeClimateReporter $(find -type f -name "*.py" ! -path "**/.venv/**") > codeclimate.json
    - pylint --exit-zero --output-format=pylint_gitlab.GitlabPagesHtmlReporter $(find -type f -name "*.py" ! -path "**/.venv/**") > public/lint/index.html
  after_script:
    - anybadge --overwrite --label $CI_JOB_NAME --value=$(cat public/badges/$CI_JOB_NAME.score) --file=public/badges/$CI_JOB_NAME.svg 4=red 6=orange 8=yellow 10=green
    - |
      echo "Your score is: $(cat public/badges/$CI_JOB_NAME.score)"
  artifacts:
    paths:
      - public
    reports:
      codequality: codeclimate.json
    when: always
  only:
    - develop
    - merge_requests
  tags:
    - docker

[Solution]
Here is a solution to resolve my problems.
I will create another config pylint for merge request event. It just check code quality on commit changed files compare with merge request target branch.

pylint_merge_request:
  stage: lint
  before_script:
    - pip install pylint-gitlab
  script:
    - echo CI_COMMIT_SHA=${CI_COMMIT_SHA}
    - echo CI_MERGE_REQUEST_TARGET_BRANCH_NAME=${CI_MERGE_REQUEST_TARGET_BRANCH_NAME}
    - git fetch origin ${CI_MERGE_REQUEST_TARGET_BRANCH_NAME}
    - FILES=$(git diff --name-only ${CI_COMMIT_SHA} origin/${CI_MERGE_REQUEST_TARGET_BRANCH_NAME} | grep '\.py'$)
    - echo "Changed files are $FILES"
    - pip install -r requirements.txt
    - pylint --rcfile=.pylintrc --output-format=text $FILES | tee /tmp/pylint.txt
  only:
    - merge_requests
  tags:
    - docker

I hope it is helpful for everyone. You can use it for other programing languages on Gitlab.

How to Free AWS Lambda Code Storage when Limit Exceeded

Cao Duy Khanh — Wed, 28 Apr 2021 21:08:52 +0000

Use the following open-source tool on GitHub.

You are already up and running with AWS Lambda for several months, and all of a sudden you get the following error:

An error occurred: TestDashdeliveryLambdaFunction – Code storage limit exceeded. (Service: AWSLambda; Status Code: 400; Error Code: CodeStorageExceededException; Request ID: 05d3ae68-e7f6-11e8-948e-41c27396380e).

Why is it happening?

AWS limits the amount of “code storage” it saves on their internal S3 for Lambda functions of up to 75GB.

Although it sounds like a lot of space, you can easily exceed that storage limit. In case you’re using the Serverless Framework, its default is to store a version for every deployment you make. It’s also affected by the programming language you are using.

Solving the issue

If you don’t need to store a version for each deployment (like many of us), you can easily cancel it with the following addition to your serverless.yml file:

provider:
  name: aws
  runtime: nodejs14.x
  versionFunctions: false
  region: ${opt:region, 'us-east-1'}
  stage: ${opt:stage, 'dev'}

Adding the versionFunctions: false parameter will cancel the version storing.

clear-lambda-storage: auto-clean old versions

If you prefer to keep older versions (e.g. for being able to rollback quickly), you’ll need to manually clean old ones. This time it’s the clear-lambda-storage. As simple as it sounds, it will take care of removing old and unused versions (i.e. that are neither currently deployed nor $LATEST) from every Lambda function, and from every region. Running it is very simple:

git clone https://github.com/khanhcd92/clear-lambda-storage-nodejs.git
cd clear-lambda-storage-nodejs/
npm i
node index.js

Advanced usage

Provide credentials:

node index.js --access-key <access_key_id> --secret-key <secret_access_key> --num-to-keep <number>

Provide credentials with one region:

node index.js --access-key <access_key_id> --secret-key <secret_access_key> --num-to-keep <number> --regions <region_code>

Provide profile:

node index.js --profile <profile_id> --num-to-keep <number>

Provide profile with one region:

node index.js --profile <profile_id> --num-to-keep <number> --regions <region_code>

How to resolve issue Access Denied between S3 and CloudFront

Cao Duy Khanh — Sun, 14 Feb 2021 03:56:05 +0000

I have setup S3 to store image files and access them through CloudFront. But I have an error Access Denied and link CloudFront was redirect to S3 link as below image:

The root cause of issue relates to “Requests made with the legacy global endpoint go to US East (N. Virginia) by default.”

Refer: https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html

To fix this issue, we need to back to setup CloudFront before.

Normally, we will select own s3 bucket which contains static files on dropdown list. In this case, endpoint is format as:<bucket>.s3.amazonaws.com
This endpoint is global endpoint. We need to change to region endpoint then issue will be fixed. The region endpoint format as: <bucket>.s3-<region>.amazonaws.com

The result as below