Forem: Alok

Understanding Multi-arch Containers, Benefits and CI/CD Integration

Alok — Wed, 14 Jun 2023 05:54:31 +0000

Have you ever seen “exec /docker-entrypoint.sh: exec format error” error message on your server while running any docker image or Kubernetes pods? This is most probably because you are running some other CPU architecture container image on your server OR did you ever use --platform linux/x86_64 option on your Apple silicon M1, M2 MacBook? If yes, then you are not getting the native performance of Apple silicon and it may be draining your MacBook battery.
To avoid this kind of error and performance issue, we need to run the correct multi-arch container image or we may need to build our own image because all container public image does not have multi-arch image available.

In this blog post, we will learn what are multi-arch container images? How it works? How to build and promote them? and we will write a sample code for building a multi-arch image in the CI/CD pipeline.

What is a Multi-arch Container Image?

A multi-arch Docker image is a list of images that has references to binaries and libraries compiled for multiple CPU architectures. This type of image is useful when we need to run the same application on different CPU architectures (ARM, x86, RISC-V, etc) without creating separate images for each architecture.

Multi-arch Container Use Cases

Performance and cost optimization: Container multi-arch is used to optimize performance on different CPU architectures. By building and deploying images that are optimized for specific architecture, we can achieve better performance and reduce resource usage. Using Karpenter we can easily deploy our workload to arm64 and get the benefit of AWS Graviton’s performance and cost savings.

Cross-platform development: If you are developing an application that needs to run on multiple platforms, such as ARM and x86, you can use buildx to build multi-arch Docker images and test the application on different architectures.

IoT devices: Many IoT/Edge devices use ARM processors, which require different binaries and libraries than x86 processors. With multi-arch images, you can create an image that can run on ARM, x86, and RISCV devices, making it easier to deploy your application to a wide range of IoT devices.

Benefits of Using Multi-arch Container Image

Several advantages of using multi-arch container images are:

Ability to run Docker image on multiple CPU architectures
Enables us to choose eco-friendly CPU architecture
Seamless migration from one architecture to another
Better performance and cost saving using arm64
Ability to support more cores per CPU using arm64

How to Build Multi-arch Container Image?

There are multiple ways to build a multi-arch container but we will be focusing on widely used and easy methods.

Traditional Docker build command
Using Docker buildx

Using Traditional Docker Build Command

In this tutorial, we will manually build both images on different CPU architecture machines and push them to the container registry (eg. Dockerhub) and then create the manifest file which has both image references. A manifest file is a simple JSON file containing the index of container images and its metadata like the size of image, sha256 digest, OS, etc. We will see more about manifest file later in this blog.

For eg. this is our basic Dockerfile.

FROM nginx
RUN echo “Hello multiarch” > /usr/share/nginx/html/index.html

########## on amd64 machine ##########
docker build -t username/custom-nginx:v1-amd64 .
docker push username/custom-nginx:v1-amd64

########## on arm64 machine ##########
docker build -t username/custom-nginx:v1-arm64 .
docker push username/custom-nginx:v1-arm64

########## Create a manifest index file ##########
docker manifest create \
    username/custom-nginx:v1 \
    username/custom-nginx:v1-amd64 \
    username/custom-nginx:v1-arm64

########## Push manifest index file ##########
docker manifest push username/custom-nginx:v1

Using Docker Buildx

With buildx, we just need to run one single command with parameterized architecture.

docker buildx build \
--push \
--platform linux/arm64,linux/amd64 \ 
-t username/custom-nginx:v1 .

In the background, the Docker buildx command uses buildkit so when we run the above command, it creates one container with moby/buildkitd image, which has QEMU binary for multiple CPU architectures which are responsible for the emulating CPU instruction sets. We can view these QEMU binaries by running ls /usr/bin/buildkit-qemu-* inside the running buildkit container.

In the above command, we passed --platform linux/arm64,linux/amd64 so it uses the /usr/bin/buildkit-qemu-aarch64 QEMU binary for building linux/arm64 image and linux/amd64 are natively built on the host machine. Once both images are built, then it uses the --push option to create the manifest file and pushes both images to the registry server with the manifest file.

By inspecting the manifest file we can see “Ref” contains the actual image link which will be fetched when platform[0].architecture matches the host system architecture.

$ docker manifest inspect -v nginx

[
        {
                "Ref": "docker.io/library/nginx:latest@sha256:bfb112db4075460ec042ce13e0b9c3ebd982f93ae0be155496d050bb70006750",
                "Descriptor": {
                        "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
                        "digest": "sha256:bfb112db4075460ec042ce13e0b9c3ebd982f93ae0be155496d050bb70006750",
                        "size": 1570,
                        "platform": {
                                "architecture": "amd64",
                                "os": "linux"
                        }
                },
                "SchemaV2Manifest": {
                        "schemaVersion": 2,
                        "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
                        "config": {
                                "mediaType": "application/vnd.docker.container.image.v1+json",
                                "size": 7916,
                                "digest": "sha256:080ed0ed8312deca92e9a769b518cdfa20f5278359bd156f3469dd8fa532db6b"
                        },
….

        {
                "Ref": "docker.io/library/nginx:latest@sha256:3be40d1de9db30fdd9004193c2b3af9d31e4a09f43b88f52f1f67860f7db4cb2",
                "Descriptor": {
                        "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
                        "digest": "sha256:3be40d1de9db30fdd9004193c2b3af9d31e4a09f43b88f52f1f67860f7db4cb2",
                        "size": 1570,
                        "platform": {
                                "architecture": "arm64",
                                "os": "linux",
                                "variant": "v8"
                        }
                },
                "SchemaV2Manifest": {
                        "schemaVersion": 2,
                        "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
                        "config": {
                                "mediaType": "application/vnd.docker.container.image.v1+json",
                                "size": 7932,
                                "digest": "sha256:f71a4866129b6332cfd0dddb38f2fec26a5a125ebb0adde99fbaa4cb87149ead"
                        }

We can also use buildx imagetools command to view the same output in a more human-readable format.

$ docker buildx imagetools inspect sonarqube:10.0.0-community

Name:      docker.io/library/sonarqube:10.0.0-community
MediaType: application/vnd.docker.distribution.manifest.list.v2+json
Digest:    sha256:51588fac6153b949af07660decfe20b5754da9fd12c82db5d95a0900b6024196

Manifests:
  Name:      docker.io/library/sonarqube:10.0.0-community@sha256:8b536568cd64faf15e1e5be916cf21506df70e2177061edfedfd22f255a7b1a0
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/amd64

  Name:      docker.io/library/sonarqube:10.0.0-community@sha256:2163e9563bbba2eba30abef8c25e68da4eb20e6e0bb3e6ecc902a150321fae6b
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/arm64/v8

If you are having any issues building multi-arch images, you can run the following command to reset the /proc/sys/fs/binfmt_misc entries.

docker run --rm --privileged multiarch/qemu-user-static --reset -p yes

We can also build multi-arch container images using Buildah as well.

How do Multi-arch Container Images Work?

As we can see in the diagram, the host machine has x86/amd64 CPU architecture, and on top of that, we install operating systems which can be Windows or Linux. Windows requires WSL or LinuxKit to run Docker. It uses QEMU to emulate multiple CPU architectures and Dockerfile builds run inside this emulation.

When we run the docker pull or build command, it fetches the requested manifest file from the registry server. These manifest files are JSON file that can have one Docker image reference or contains more than one image list. It fetches the correct image depending on the host machine's CPU architecture.

How to Integrate Multi-arch Container Build with CI/CD?

If your workload runs on multiple machines with different CPU architectures, it is always better to build multi-arch Docker images for your application. Integrating multi-arch build into CI/CD streamlines the image build and scan process easier, adds only one Docker tag, and saves time. Below we have written Jenkins and GitHub CI sample code for building multi-arch images.

Jenkins Multi-arch CI

Currently, the Jenkins Docker plugin does not support multi-arch building so we can use buildx to build multi-arch images.

pipeline
{
    agent {
        label 'worker1'
    }

    options{
        timestamps()
        timeout(time: 30, unit: 'MINUTES')
        buildDiscarder(logRotator(numToKeepStr: '10'))
    }
    environment {
        DOCKER_REGISTRY_PATH = "https://registry.example.com"
        DOCKER_TAG = "v1"
    }


    stages
    {
        stage('build-and-push')
        {
        steps{
          script{
            docker.withRegistry(DOCKER_REGISTRY_PATH, ecrcred_dev){
            sh '''
              ####### check multiarch env ###########
              export DOCKER_BUILDKIT=1
              if [[ $(docker buildx inspect --bootstrap | head -n 2 | grep Name | awk -F" " '{print $NF}') != "multiarch" ]]
              then
                docker buildx rm multiarch | exit 0
                docker buildx create --name multiarch --use
                docker buildx inspect --bootstrap
              fi
              ####### Push multiarch ###########
              docker buildx build --push --platform linux/arm64,linux/amd64 -t "$DOCKER_REGISTRY_PATH"/username/custom-nginx:"$DOCKER_TAG" .
            '''
            }
          }
        }
      }
    }
}

Otherwise, we can use the traditional Docker build command as shown above in Jenkins stages with different sets of Jenkins worker nodes.

GitHub CI Pipeline for Building Multi-arch Container Images

GitHub Actions also supports multi-arch container images. It also uses QEMU CPU emulation in the background.

name: docker-multi-arch-push
on:
  push:
    branches:
      - 'main'
jobs:
  docker-build-push:
    runs-on: ubuntu-20.04
    steps:
      - name: Checkout Code
        uses: actions/checkout@v3


      - name: Set up QEMU
        uses: docker/setup-qemu-action@v2


      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2


      - name: Login to docker.io container registry
        uses: docker/login-action@v2
        with:
          username: ${{ secrets.DOCKER_USER }}
          password: ${{ secrets.DOCKER_TOKEN }}


      - name: Build and push
        id: docker_build
        uses: docker/build-push-action@v3
        with:
          context: .
          file: ./Dockerfile
          platforms: linux/amd64,linux/arm64
          push: true
          tags: username/custom-nginx:latest

How to Promote your Multi-arch Image to Higher Environments?

Promoting Docker multi-arch requires a few extra steps as the docker pull command only pulls a single image based on the host machine's CPU architecture. To promote multi-arch Docker images we need to pull all CPU architecture images one by one by using –plarform=linux/$ARCH and then create a manifest file and push them to the new registry server. To avoid these complex steps we can leverage the following tools.

Skopeo or Crane can be used to promote our multi-arch image from one account to another using just a single command. In the background, what these tools do is use Docker API to fetch all multi-arch images and then create manifest and push all images and manifest.

$ skopeo login --username $USER docker.io

$ skopeo copy -a docker://dev-account/custom-nginx:v1 docker://prod-account/custom-nginx:v1

What if you want to use only the Docker command to promote this image to a higher environments (Production)?

####### Pull DEV images ###########
docker pull --platform=amd64 "$DOCKER_IMAGE_NAME_DEV":"$DOCKER_TAG"
docker pull --platform=arm64 "$DOCKER_IMAGE_NAME_DEV":"$DOCKER_TAG"


####### Tag DEV image with STAGE ###########
docker tag "$DOCKER_IMAGE_NAME_DEV":"$DOCKER_TAG" "$DOCKER_IMAGE_NAME_STAGE":"$DOCKER_TAG"-amd64


docker tag "$DOCKER_IMAGE_NAME_DEV":"$DOCKER_TAG" "$DOCKER_IMAGE_NAME_STAGE":"$DOCKER_TAG"-arm64


####### Push amd64 and arm64 image to STAGE ###########
docker push "$DOCKER_IMAGE_NAME_STAGE":"$DOCKER_TAG"-amd64
docker push "$DOCKER_IMAGE_NAME_STAGE":"$DOCKER_TAG"-arm64


####### Create mainfest and push to STAGE ###########
docker manifest create \
    "$DOCKER_IMAGE_NAME_STAGE":"$DOCKER_TAG" \
    --amend "$DOCKER_IMAGE_NAME_STAGE":"$DOCKER_TAG"-amd64 \
    --amend "$DOCKER_IMAGE_NAME_STAGE":"$DOCKER_TAG"-arm64


docker manifest push "$DOCKER_IMAGE_NAME_STAGE":"$DOCKER_TAG"

How to Scan Multi-arch Images for Vulnerabilities?

We can use any tool like Trivy, Gryp, or Docker scan for image scanning but we have to pull multi-arch images one by one and then scan them because by default Docker pull command will only fetch the one image that matches with the Host CPU. We can leverage the Docker pull command with --platform={amd64, arm64} to pull different CPU arch images. Here is how we can do that:

####### Pull amd64 image and scan ###########
docker pull --platform=amd64 nginx:latest
trivy image nginx:latest


####### Pull arm64 image and scan ###########
docker pull --platform=arm64 nginx:latest
trivy image nginx:latest

Some Caveats of Using Multi-arch Container

There are prominent benefits of using multi-arch containers, but there are some caveats that you should certainly be aware of before taking the leap.

It takes extra storage for storing other arch images.
Takes time to build the multi-arch container image also while building the arm64 on QEMU emulation consumes lots of time and resources.
Performance is significantly slower with emulation to run binaries on different CPUs compared to running the binaries natively.
There are still some issues with buildx building arm64 images like the base image not being available in arm64 and also performing sudo level access or building cross-compile statically linked binary requires an extra step.
Need container scanning for all images one by one.
Buildx multi-arch builds are only supported on amd64 CPU architecture.

Conclusion

In this blog, we saw what are multi-arch containers and their use cases. We integrated multi-arch build with Jenkins and Github CI with sample code and provides you the several ways to promote and scan multi-arch container images, and finally, we learned the caveats of using multi-arch containers.

Using multi-arch images gives us the ability to build once and run everywhere. We can seamlessly migrate from one CPU arch to another CPU with ease. Also by deploying images that are optimized for specific architectures, we can achieve better performance and reduce resource costs.

Thanks for reading this post. We hope it was informative and engaging for you. We would love to hear your thoughts on this post, so do start a conversation on LinkedIn.

Looking for help with building your DevOps strategy or want to outsource DevOps to the experts? Learn why so many startups & enterprises consider us as one of the best DevOps consulting & services companies.

Refrences

Implement DevSecOps to Secure your CI/CD pipeline

Alok — Wed, 28 Sep 2022 06:16:37 +0000

Before understanding DevSecOps, let’s understand what is DevOps. DevOps is the combination of cultural philosophies, practices, and tools that increase an organization’s ability to deliver applications and services at high velocity.

In fast-moving projects, security often lags behind and given low priority which may lead to buggy code and hacks. Let’s see how we can reduce the risk of attack by integrating security in our DevOps pipeline.

What is to DevSecOps (DevOps + Security)

DevSecOps is a cultural approach where every team and person working on an application considers security throughout it's lifecycle. It ensures that security is implemented at every stage of the application software development lifecycle (SDLC) by incorporating required security checks embedded into CI/CD automation using appropriate tools.

For example, let's see how the DevSecOps process can detect and prevent zero-day vulnerabilities like log4j. Using Syft tool, we can generate SBOM for our application code and pass this SBOM report to Grype which can detect these new vulnerabilities and report to us if there is any fix or patch available. As these steps are part of our CI/CD, we can alert our developers and security team to remediate this issue as soon as it is identified.

What are the benefits of using DevSecOps?

Finds vulnerability and bugs at an earlier stage of development
Streamlined compliance
Speedy recovery
Secure supply chain
Cost saving
Can include AI-based monitoring for detecting anomalies
Reduces the risk of surface attack and increases confidence
Full visibility of potential threats and possible ways to remediate it

How to make security culture your default state?

Unless you’ve included security in every employee’s onboarding, creating a widespread security culture mindset will be challenging. Employees will need to think differently, behave differently, and eventually turn those changes into habits so that security becomes a natural part of their day-to-day work.

Read more about Gitlab's DevSeOps security culture.

What does DevSecOps CI/CD pipeline look like?

Jenkins DevSecOps pipeline

In this post, we will cover the following standard CI/CD stages and how to secure them:

Plan/Design
Develop
Build and Code analysis
Test
Deploy
Monitor and Alert

Alright, let's dive into how to implement DevSecOps!

1. Plan/Design

In this stage, we outline where, how, and when integration, deployment, and security testing will be done.

1.1 Threat modeling:

It effectively puts you in the mindset of an attacker and allows us to see the application through the attacker's eyes and block their attack before they get a chance to do anything about it. We can use OWASP threat modeling or Simple questions method from Microsoft to design our threat modeling. We can also use OWASP Threat Dragon and Cairis open source threat modeling tools to create threat model diagrams for our secure development lifecycle.

1.2 Secure SDLC

A Secure SDLC requires adding security testing at each software development stage, from design to development, to deployment, and beyond. Examples include designing applications to ensure that your architecture will be secure, as well as including security risk factors as part of the initial planning phase.

- Snyk Secure SDLC

In a secure software development cycle, we should separate our development, testing, and production environments and also have authorization processes that control the deployment promotion from one environment to another. This reduces the risk of a developer making unauthorized changes. This ensures that any modifications pass through a standard approval process.

2. Develop

The Development stage starts with writing code and we can use shift-left security best practice which incorporates security thinking in the earliest stages of development. For example:

Install linting tools inside the code editor like Visual Studio Code. One of the most popular linting tools is SonarLint. Which highlights bugs and security vulnerabilities as you write code.
Use Pre-commit hooks to prevent adding any secrets to code.
Setup Protected branch and code reviews process.
Sign git commit with GPG key.
Always verify the downloaded binary/file hash.
Enable 2-factor authentication.

3. Build and code analysis

Before building, we need to scan our code for vulnerabilities and secrets. By doing static code analysis, it may detect a bug or a possible overflow in code and these overflows lead to a memory leak, which degrades the system performance by reducing the amount of memory available for each program. Sometimes it can be used as an attack surface by hackers to exploit the data.

3.1 Scan for secrets and credentials

detect-secret is an enterprise-friendly tool for detecting and preventing secrets in the code base. We can also scan the non-git tracked files. There are other tools as well like Gitleaks which also provide similar functionality.

detect-secrets scan test_data/ --all-files

3.2 Software Bill of Materials (SBOM)

SBOM lets us Identify all software components, libraries, and modules that are running in our environment, even their dependencies. It speeds up response time for new vulnerabilities - including zero-day vulnerabilities like Log4j.

We can use below tools for SBOM report.

3.2.1 Syft with Grype and Trivy

Syft tool gives container image and filesystem SBOM result in CycloneDX open source format which can be shared easily. Syft also supports cosign attestations for verifying legit images.

syft nginx:latest -o cyclonedx-json=nginx.sbom.cdx.json

So we have generated an SBOM report that shows what libs and modules are running in our software. Now, let's scan for vulnerabilities in SBOM reports using Grype.

[root@laptop ~]# grype sbom:./nginx.sbom.cdx.json | head
 ✔ Vulnerability DB       [no update available]
 ✔ Scanned image          [157 vulnerabilities]
NAME            INSTALLED           FIXED-IN        TYPE  VULNERABILITY     SEVERITY   
apt             2.2.4                                       deb   CVE-2011-3374     Negligible  
bsdutils        1:2.36.1-8+deb11u1                      deb   CVE-2022-0563     Negligible  
coreutils       8.32-4+b1                                   deb   CVE-2017-18018    Negligible  
coreutils       8.32-4+b1           (won't fix)     deb   CVE-2016-2781     Low      
curl            7.74.0-1.3+deb11u1                      deb   CVE-2022-32208    Unknown  
curl            7.74.0-1.3+deb11u1                      deb   CVE-2022-27776    Medium   
curl            7.74.0-1.3+deb11u1  (won't fix)     deb   CVE-2021-22947    Medium   
curl            7.74.0-1.3+deb11u1  (won't fix)     deb   CVE-2021-22946    High     
curl            7.74.0-1.3+deb11u1  (won't fix)     deb   CVE-2021-22945    Critical  

# Or we can directly use Grype for SBOM scanning
grype nginx:latest

Note: - Many of the vulnerabilities that are scanned by SCA tools are neither exploitable nor fixable via regular updates. curl and glibc are some examples. So these tools show them as not fixable or won't fix.

The latest version of Trivy can also generate SBOM reports, but it's mostly used for finding vulnerabilities in containers and filesystems.

3.2.2 OWASP Dependency-Check

OWASP Dependency-Check a Software Composition Analysis (SCA) tool that attempts to detect publicly disclosed vulnerabilities contained within a project's dependencies. It does this by determining if there is a Common Platform Enumeration (CPE) identifier for a given dependency. If found, it will generate a report linking to the associated CVE entries. We can also publish our SBOM report to Dependency-Track and visualize our software components and their vulnerabilities.

dependency-check.sh --scan /project_path

Once we know what type of vulnerabilities are present in our software, we can patch them and make our application safe and secure.

3.3 Static Application Security Testing (SAST)

It's a method of debugging code without running the program. It analyzes the code based on predefined rule sets.

SonarQube allows all developers to write cleaner and safer code. It supports lots of programming languages for scanning (Java, Kotlin, Go, JavaScript). It also supports running unit testing for code coverage. It can be easily integrated with Jenkins and Azure DevOps. Checkmarx, Veracode, and Klocwork also provide similar functionality but these are paid tools.

docker run \
    --rm \
    -e SONAR_HOST_URL="http://${SONARQUBE_URL}" \
    -e SONAR_LOGIN="AuthenticationToken" \
    -v "${YOUR_REPO}:/usr/src" \
    sonarsource/sonar-scanner-cli

Source: Running SonarScanner from the Docker image.

3.4 Unit test

In Unit tests, individual software code components are checked if it is working as expected or not. Unit tests isolate a function or module of code and verify its correctness. We can use tools like JaCoCo for Java and Mocha, and Jasmine for NodeJS to generate unit test reports. We can also send these reports to SonarQube which shows us code coverage and the percentage of your code covered by your test cases.

Once SAST is done, we can scan our Dockerfile as well.

3.5 Dockerfile static scanning

Always scan the Dockerfile for vulnerabilities as while writing Dockerfile we may miss some of the best practices which may lead to vulnerable containers. To name a few common mistakes that we can avoid.

Do not use the latest docker image tag
Ensure that a user for the container has been created

Checkov or docker scan can be used to scan Dockerfile which follows best practice rules to write Dockerfile.

docker run -i -v $(pwd):/output bridgecrew/checkov -f /output/Dockerfile -o json

After building a container image, we scan it for vulnerabilities and sign our container image.

3.6 Container image scan

Scanning images gives the security state of the container images and let us take actions that result in a more secure container image. We should avoid installing unnecessary packages and use a multistage method. This keeps the image clean and safe. Scanning of images should be done in both development and production environments.

Below are a few well-known open-source and paid tools that we can use for container scanning:

Open source: Trivy, Gryp and Clair are widely used open source tools for container scanning.
Docker scan: It uses Snyk as the backend engine for scanning. It can also scan Dockerfile.
Aqua scan: Provides container image scanning but it has one unique feature Aqua DTA (Dynamic Threat Analysis) for containers which monitors behavioral patterns and Indicators of Compromise (IoCs) such as malicious behavior and network activity, to detect container escapes, malware, cryptocurrency miners, code injection backdoors and additional threats.

trivy image nginx:latest
# OR
docker scan nginx:latest

3.7 Container image signing and verifying

If the container build process is compromised, it makes users vulnerable to accidentally using the malicious image instead of the actual container image. Signing and verifying the container always ensure we are running the actual container image.

Using distroless images not only reduces the size of the container image it also reduces the surface attack. The need for container image signing is because even with the distroless images there is a chance of facing some security threats such as receiving a malicious image. We can use cosign or skopeo for container signing and verifying. You can read more about securing containers with Cosign and Distroless Images in this blog.

cosign sign --key cosign.key custom-nginx:latest
cosign verify -key cosign.pub custom-nginx:latest

3.8 Container image validation test

Adding an extra layer of security on the container image to verify if it is working as expected and has all required files with correct permissions. We can use dgoss to do validation tests of container images.

For example, let's do a validation test for the nginx image that is running on port 80, has internet access, and verifies the correct file permission of /etc/nginx/nginx.conf, and the nginx user shell in the container.

dgoss edit nginx
goss add port 80
goss add http https://google.com
goss add file /etc/nginx/nginx.conf
goss add user nginx

# Once we exit it will copy the goss.yaml from the container to the current directory and we can modify it as per our validation.
# Validate
[root@home ~]# dgoss run -p 8000:80 nginx
INFO: Starting docker container
INFO: Container ID: 5f8d9e20
INFO: Sleeping for 0.2
INFO: Container health
INFO: Running Tests
Port: tcp:80: listening: matches expectation: [true]
Port: tcp:80: ip: matches expectation: [["0.0.0.0"]]
HTTP: https://google.com: status: matches expectation: [200]
File: /etc/nginx/nginx.conf: exists: matches expectation: [true]
File: /etc/nginx/nginx.conf: mode: matches expectation: ["0644"]
File: /etc/nginx/nginx.conf: owner: matches expectation: ["root"]
File: /etc/nginx/nginx.conf: group: matches expectation: ["root"]
User: nginx: uid: matches expectation: [101]
User: nginx: gid: matches expectation: [101]
User: nginx: home: matches expectation: ["/nonexistent"]
User: nginx: groups: matches expectation: [["nginx"]]
User: nginx: shell: matches expectation: ["/bin/false"]
Total Duration: 0.409s
Count: 13, Failed: 0, Skipped: 0
INFO: Deleting container

We can also use kgoss to do validation tests on pods.

Till now we have built and scanned the container image but before deploying let's test and scan the deployment or Helm chart.

4. Test

Testing ensure that the application is working as expected and has no bug or vulnerabilities.

4.1 Smoke test

Smoke tests are small but check critical components and functionality of the application. When implemented, It runs on every application build to verify critical functionality passes before integration and end-to-end testing can take place which can be time-consuming. Smoke tests help create fast feedback loops that are vital to the software development life cycle.

For example, in a smoke test, we can run the curl command on API to get the HTTP response code and latency.

4.2 API testing

Today's applications might expose hundreds of highly valuable endpoints that are very appealing to hackers. Ensuring your APIs are secure before, during, and after production is crucial. Hence we need to test our APIs.

API Testing reports what type of authentication is required and whether sensitive data is encrypted over HTTP and SQL injections allowing you to bypass the login phase.

We can use Jmeter, Taurus, Postman, and SoapUI tools for API testing. Below is a small example using Jmeter where test.jmx contains the API test cases.

jmeter -n --t test.jmx -l result.jtl

4.3 Dynamic application security testing (DAST)

DAST is a web application security test that finds security issues in the running application. DAST tools are also known as web application vulnerability scanners which can detect common vulnerabilities like SQL injection, cross-site scripting, security misconfigurations, and other common issues detailed in OWASP Top 10. We can use HCL Appscan, ZAP, Burp Suite, and Invicti which finds vulnerabilities in the running web application. Here is a list of DAST scanning tools provided by OWASP. We can easily integrate these tools with our CI/CD pipeline.

zap.sh -cmd -quickurl http://example.com/ -quickprogress -quickout example.report.html

5. Deploy

Deployment can be of infrastructure or application; however, we should scan our deployment files. We can also add a manual trigger where the pipeline waits for external user validation before proceeding to the next stage, or it can be an automated trigger.

5.1 Static scan of Kubernete manifest file or Helm chart

It is always a good practice to scan your Kubernetes deployment or Helm chart before deploying. We can use Checkov to scans Kubernetes manifests and identifies security and configuration issues. It also supports Helm chart scanning. We can also use terrascan and kubeLinter to scan the Kubernetes manifest.

docker run -t -v $(pwd):/output bridgecrew/checkov -f /output/keycloak-deploy.yml -o json
# For Helm
docker run -t -v $(pwd):/output bridgecrew/checkov -d /output/ --framework helm -o json

5.2 Pre-deploy policy check Kubernete manifest YAML file

Kyverno adds an extra layer of security where only the allowed type of manifest is deployed onto kubernetes, otherwise, it will reject or we can set validationFailureAction to audit which only logs the policy violation message for reporting. Kubewarden and Gatekeeper are alternative tools available to enforce policies on Kubernetes CRD.

Here is a simple Kyverno policy to disallow the image's latest tag.

5.3 kube-bench for CIS scan

kube-bench checks whether Kubernetes is deployed securely by running the checks documented in the CIS Kubernetes Benchmark. We can deploy kube-bench as a Job that runs daily and consume its report in CI/CD to pass or fail the pipeline based on the level of severity.

kubectl apply -f eks-job.yaml
kubectl logs kube-bench-pod-name

5.4 IaC scanning:

Checkov, Terrascan, and Kics can be used to scan our Infrastructure code. It supports Terraform, Cloudformation, and Azure ARM resources.
Terratest can be used to test infrastructure in real-time.

terraform init
terraform plan -out tf.plan
terraform show -json tf.plan | jq '.' > tf.json
checkov -f tf.json

After scanning for Kubernetes deployment and kube-bench we can deploy our application and start with the testing stage.

6. Monitoring and Alerting

Monitoring and alerting is the process of collecting logs and metrics about everything happening in our infrastructure and sending notifications based on the metrics threshold value.

6.1 Metrics monitoring

Prometheus: It's a widely used open source tool for metrics monitoring. It provides various exporters that can be used for monitoring systems or application metrics. We can also use Grafana to visualize prometheus metrics.
Nagios and Zabbix: These are open source software tools to monitor IT infrastructures such as networks, servers, virtual machines, and cloud services.
Sensu Go: It is a complete solution for monitoring and observability at scale.

6.2 Log monitoring

OpenSearch/Elasticsearch: It is a real-time distributed and analytic engine that helps in performing various kinds of search operations.
Graylog: It provides centralized log management functionality for collecting, storing, and analyzing data.
Grafana Loki: It's a lightweight log aggregation system designed to store and query logs from all your applications and infrastructure.

6.3 Alerting

Prometheus Alertmanager: The Alertmanager handles alerts sent by client applications such as the Prometheus server.
Grafana OnCall: Developer-friendly incident response with phone calls, SMS, slack, and telegram notifications.

Security-focused logging and monitoring policy is used to prevent sensitive information from being logged in plain text. We can write a test case in our logging system to look for certain patterns of data. For example, a regex to find out sensitive information so that we can detect the logs in a lower environment.

Application performance Monitoring (APM) improves the visibility into a distributed microservices architecture. The APM data can help enhance software security by allowing a full view of an application. Distributed tracing tools like Zipkin and Jaeger kind of stitch all logs together and bring full visibility of requests from start to end. It speeds up response time for new bugs or attacks.

Although all cloud providers have their own monitoring toolsets and some tools are accessible from the marketplace. Also, there are paid monitoring tool providers like Newrelic, Datadog, Appdynamics, and Splunk that provide all types of monitoring.

6.4 Security information and event management (SIEM)

Security information and event management (SIEM) offer real-time monitoring and analysis of events as well as tracking and logging of security data for compliance or auditing purposes. Splunk, Elastic SIEM, and Wazuh which give automated detection of suspicious activity and tools with behavior-based rules also can detect anomalies using prebuilt ML jobs.

6.5 Auditing

After the deployment visibility comes from the level of auditing that has been put in place on application and infrastructure. The goal would be to have your auditing at a level that allows you to feed info into a security tool to give needed data. We can enable audits on the AWS cloud using CloudTrail and on Azure with platform logs. For auditing applications, we can enable inbuilt audit logs and send the audit data to any logging tool like Elasticseach using auditbeat or Splunk and create an auditing dashboard.

6.6 Kubernetes runtime security monitoring

Falco is a cloud native Kubernetes threat detection tool. It can detect unexpected behavior, intrusions, and data theft in real time. In the backend, it uses Linux eBPF technology to trace your system and applications at runtime. For example, it can detect if someone tries to read a secret file inside a container, access a pod as a root user, etc, and trigger a webhook or send logs to the monitoring system. There are similar tools like Tetragon, KubeArmor, and Tracee which also provide Kubernetes runtime security.

Till now, we have seen how DevSecOps CI/CD pipeline looks like. Now, let's dive into adding more security layer on top.

Best practices to secure your infrastructure for DevSecOps CI/CD

Network Security

Networking is our first defense against any kind of attack and to prevent attacks on our application we should harden our network.

Create a separate private network for the workload (eg. App and DB) and only allow internet access from NAT.
Set fine-grained access on inbound and outbound network rules. Also, we can use Cloud custodian to set the security compliance which automatically removes any unwanted network traffic.
Always configure Network ACL (NACL) for subnets in AWS. The best practice would be to block all outbound traffic and then allow the required rules.
Use Web Application Firewall (WAF).
Enable DDOS protection.
Nmap and Wireshark, tcpdump tools can be used to scan networks and packets.
Use VPN or Bastion host for connecting to infrastructure networks.

Web Application Firewall (WAF)

WAF is a layer7 firewall that protects our web applications against common web exploits (like XSS and SQL injection) and bots that may affect availability, compromise security, or consume excessive resources. Most cloud service provider provides WAF and with a few clicks, we can easily integrate it with our application.

Curiefense is an open source cloud native self-managed WAF tool that can be used to protect all forms of web traffic, services, DDoS, and APIs. We can also use WAF as a service from Cloudflare and Imperva.

Identity Access Management (IAM)

IAM is a centrally defined policy to control access to data, applications, and other network assets. Below are a few methods that help prevent unauthorized access.

Have centralized user management using Active Directory or LDAP.
Use RBAC access management.
Have fine-grained access Policy for AWS IAM role.
Rotate user's access and secret keys periodically.
Use Teleport for centralized connectivity, authentication, authorization, and audit.
Store secrets into vaults and ensure that it is only accessible to authorized users.
Implement Zero trust within your services.

Cloud, server, and application hardening

We can use CIS benchmark to harden cloud, operating system, and application. It is always a good practice to use a hardened OS as it reduces the attack surface of the server. Most of the cloud providers provide a hardened image or we can create our own custom hardened image.

Nowadays, most applications run inside containers. We need to harden our applications as well as containers by doing static analysis and container image scanning.

To protect against viruses, trojans, malware, and other malicious threats we can install Antivirus like Falcon, SentinelOne, or Clamav.

Server patching

The most common attack vector exploits vulnerabilities in the OS or applications running on servers. Running regular vulnerability scans against the environments and updating regular packages reduces the risk of vulnerability.

We can create an automation pipeline to patch the server using Foreman or Red Hat Satellite and for scanning, we can use OpenVAS or Nessus to get the list of vulnerabilities.

Securing Kubernetes

Kubernetes has become the backbone of modern infrastructure and to make sure we are running it securely we can use the below tools:

Use correct security-context in Kubernetes YAML file.
Use Network Policy to block all the traffic by default and allow only required traffic.
Use Service Mesh (Linkerd, Istio) to have mTLS communication between microservices and implement Authorization to have fine-grained access.
Implement kube-bench for a CIS benchmark report for the kubernetes cluster. We can run this scan daily in our Kubernete cluster and fix any reported vulnerabilities.
Use tool like Kube-hunter, Popeye and Kubescape for security weaknesses and misconfigurations in kubernetes clusters and visibility of security issues.
Use Checkov, KubeLinter, and Terrascan for scanning Kubernete YAML, Helm chart with best practices and vulnerabilities.
Implement pre-deployment policy checks like Kyverno, Kubewarden, and Gatekeeper can block non-standard Deployment.
Use a hardened image for the worker server. All cloud providers provide CIS benchmark harden images. We can also build our own custom hardened image using amazon-eks-ami.
Store Kubernetes secret in an encrypted format or use an external secret manager like Vault.
Use IAM roles for service accounts to assign AWS roles directly to Kubernete service accounts.
Implement Chaos Mesh and Litmus chaos engineering framework to understand the behavior and stability of application in real-world use cases.
Follow best practice to secure kubernetes
Use tool like Falco, Tracee to monitor runtime privileged and unwanted system calls.

Containers

Containers are the smallest level of abstraction for running any workload in modern infrastructure. Below are a few methods to secure our containers and we have also seen above how to integrate them into our CI/CD pipeline.

Scan the Container image and Dockerfile.
Reduce the Docker image size with a multi-stage build and using a distroless image to reduce the attack surface.
Don't use the root user and privileged containers.
Have Gvisor and Kata containers for kernel isolation.
Use container image signing and verification.
Set up a list of known good containers registry.
Implement the Container security best practices.

Software supply chain security

Supply chain security is crucial to the overall security of a software product. An attacker who can control a step in the supply chain can alter the product for malicious intents that range from introducing backdoors in the source code to including vulnerable libraries in the final product.

- in-toto Specification

As per Anchore 2022 Software Supply Chain Security Report 62% of Organizations Surveyed have been Impacted by Software Supply Chain Attacks. Implementing DevSecOps CI/CD significantly reduces the Supply chain attack as we scan all our software components like Code, SBOM, Containers, infrastructure, Sign and verify containers, etc.

Center for Internet Security (CIS) standards

CIS is a non-profit organization that provides security standards benchmark to secure our infrastructure. One of the benefits of following the CIS benchmark is that it directly maps to several established standards guidelines including the NIST Cybersecurity Framework (CSF), ISO 27000 series of standards, PCI DSS, HIPAA, and others. Also, Level 2 CIS benchmark gives more security controls.

Vulnerability Assessment & Penetration Testing (VAPT)

VAPT is a security testing method used by organizations to test their applications, network, endpoint, and cloud. It can be performed by internal and external third-party vendors. Depending upon compliance and regulation also how risky the technology is, organizations do schedule VPAT scans quarterly, half-yearly, or annually.

Vulnerability Assessment

A Vulnerability Assessment (VA) is a security process to identify weaknesses or vulnerabilities in an application, system, and network. The goal of a vulnerability assessment is to determine all vulnerabilities and help the operator fix them. DAST scanning is also part of vulnerability assessment and it's often quick ranging from 10 minutes to 48 hours depending on the configuration. It is easier to integrate with our CI/CD pipeline whereas pen testing goes beyond VA and has aggressive scanning and exploitation after discovering any vulnerabilities.

Penetration Testing

Pen testing is a proactive cybersecurity practice where security experts target individual components or whole applications to find vulnerabilities that can be exploited to compromise the application and data. ZAP, Metasploit, and Burp Suite can be used for doing pen tests and it can discover vulnerabilities that might be missed by SAST and DAST tools. The downside of a pen test is that it takes more time depending on the coverage and configuration. The proper pen test might take up to several weeks, and with DevOps development speed, it becomes unsustainable. However, it's still worth adding Internal VAPT which can be done on every feature release to move fast and external VAPT can be done biannually or annually to keep overall security in check.

Conclusion

To quickly recap what we have done to make the DevSecOps pipeline, we scanned for secrets, SAST and SBOM to find any vulnerabilities in our code. After that, we scanned our Dockerfile, container image, Kubernete manifest and did a container validation test, and signed our container image to ensure it's safe and secure. After deployment, we did Smoke, API test, and DAST scanning to ensure there is no bug in deployment. Always remember security requires constant attention and improvement. However, these can be the first few steps on the never-ending journey towards DevSecOps.

Implementing DevSecOps best practices reduces the risk of vulnerabilities and hacking. Scanning all parts of your infrastructure and application gives full visibility of potential threats and possible ways to remediate them. "The only way to do security right is to have multiple layers of security" hence we talked about multiple methods and tools that can be used for finding vulnerabilities.

Here is a list of a few tools that we have used to set up our DevSecOps pipeline.

Category	Tools
Threat modeling	Threat dragon, Cairis
Secret scan	detect-secret, Gitleaks, git-secrets
SBOM scan	Syft, Grype, Trivy, Dependency-check, Dependency-track
SAST scan	SonarQube, Checkmarx, Veracode, Klocwork
Unit testing	JaCoCo, Mocha, Jasmine
Dockerfile scan	Checkov, docker scan
Container scan	Trivy, Grype, Clair, docker scan, Aqua scan
Container signing	Cosign, Skopeo
Container validation	goss, kgoss
Kubernete manifest scan	Checkov, Terrascan, KubeLinter
Kubernetes manifest pre-check	Kyverno, Kubewarden, Gatekeeper
CIS scan	kube-bench, CIS-CAT Pro, Prowler
IaC scan	Checkov, Terrascan, KICS, Terratest
API testing	JMeter, Taurus, Postman, SoapUI
DAST scan	ZAP,HCL Appscan, Burp Suite, Invicti, Checkmarx, InsightAppSec
Distributed tracing	Zipkin, Jaeger
Cloud native runtime security	Falco, Tetragon, Kubearmor, Tracee
Service mesh	Istio, Linkerd, Cilium, Traefik
Network security scan	Nmap, Wireshark, tcpdump, OpenVAS, Metasploit
Antivirus scan	Falcon, SentinelOne, Clamav
OS vulnerability scan	OpenVAS, Nessus, Nexpose
OS patching	Foreman, Red Hat Satellite, Uyuni
Pen testing	ZAP, Metasploit, Burp Suite

That's a wrap folks :) Hope the article was informative and you enjoyed reading it. For more posts like this one, do subscribe to our weekly newsletter. I'd love to hear your thoughts on this post, so do start a conversation on LinkedIn :)

Looking for help with DevSecOps? do check out our capabilities and how we’re helping startups & enterprises as an DevSecOps consulting services provider.

References & further reading

Implementing Cloud Governance as a Code using Cloud Custodian

Alok — Thu, 09 Dec 2021 10:45:38 +0000

In today’s scaling cloud infrastructure it's hard to manage all resources compliance. Every organization has a set of policies to follow for detecting violations and taking remediation actions on their cloud resources. This is generally done by writing multiple custom scripts and using some 3rd party tool and integration. Many development teams know how hard it is to manage and write custom scripts and keep a track of those. This is where we can leverage Cloud Custodian DSL policies to manage our Cloud resources with ease.

What is cloud governance?

Cloud governance is a framework which defines how developers can create policies to control costs, minimize security risks, improve efficiency and accelerate deployment.

What are other tools that provide governance as code?

AWS Config

AWS config records and monitors all configuration data of AWS resources and We can build rules to help us enforce compliance. Setting up a Multi account and Multi Zone option is available. It also provides some predefined AWS managed rule that we can use or we can write our own custom rules. We can also take remediation action based on matches. For Custom policy we need to write our own lambda function for taking action.

However we can use Cloud Custodian to set up AWS Config rule and Custom rule which supports Multi account and Multi region using c7n-org. Also it can automatically provision aws lambda function.

Azure Policy

Azure policy enforces organization standards across Azure resources. It provides an aggregated view to evaluate the overall state of the environment, with the ability to drill down to the per-resource, per-policy granularity.(eg. Users are only allowed to create A and B series Virtual Machines). We can turn on in-built policies or create custom policies for all resources. It can also take auto remediation action on non-compliant resources.

Azure Policy is reliable and efficient for building a custom validation layer on deployments to prevent deviation from customer defined rules. Cloud Custodian and Azure Policy have significant overlap in scenarios they can accomplish with regard to compliance implementations. When reviewing your requirements, we recommend first identifying the requirements that can be implemented via Azure Policy. Custodian can then be used to implement the remaining requirements. Custodian is also frequently used to add a second layer of protection or mitigation actions to requirements covered by Azure Policy. This way we can ensure that policy is configured correctly.

— Azure Policy Comparison

Till now, we have seen What is cloud governance and what are other tools available in the market. Let's see now what Cloud Custodian can provide us in cloud governance.

What is Cloud Custodian?

Cloud Custodian is CNCF sandbox project for governing public cloud resources in real-time. It helps us write governance as code the same way we write infrastructure as code. It detects the non-complaints resource and takes action to remediate it. Custodian is a cloud native tool. It can be used with multiple cloud providers(AWS, AZURE, GCP, etc)

We can use Cloud Custodian as below,

Compliance and Security as code - We can write Simple YAML DSL policy as a code.
Cost savings - Removing unwanted resources and Implementing the on/off hours policy can save costs.
Operational efficiency -By adding governance as code it reduces the friction for innovating securely in the cloud and also increases developer efficiency.

How does it work?

When we run Cloud Custodian command depending on the Cloud provider it takes resources, filters, action as input and translate into Cloud provider API Call(eg. AWS Boto3 API). No need to worry about custom script or aws cli commands. We get clean, readable policies and numerous common filters and actions that have been built into Cloud Custodian. If we need custom filters we can always use JMESPath to write our filter.

There can be situations where we may need to run our policy periodically or based on some events. For this Cloud Custodian automatically provision lambda function and CloudWatch event rule. CloudWatch event rules can be scheduled (every 10 minutes) or triggered in response to API calls by CloudTrail, EC2 instance state events, etc.

How to install and set up Cloud Custodian ?

We can simply install Cloud Custodian with python pip command

python3 -m venv custodian
source custodian/bin/activate
pip install c7n       # This includes AWS support
pip install c7n_azure # Install Azure package
pip install c7n_gcp   # Install GCP Package

Using Cloud Custodian docker image

docker run  -it \
  -v $(pwd)/output:/opt/custodian/output \
  -v $(pwd)/policy.yml:/opt/custodian/policy.yml \
  --env-file <(env | grep "^AWS\|^AZURE\|^GOOGLE|^KUBECONFIG") \
     cloudcustodian/c7n run -v --cache-period 0 -s /opt/custodian/output /opt/custodian/policy.yml

Note: ACCESS and SECRET KEY, DEFAULT_REGION and KUBECONFIG are fetched from ENV variables and users should have access to required IAM Roles and Policies that we define in policy YAML file. Another option is to mount the file/directory inside the container.

Cloud Custodian policy.yaml explained

Cloud Custodian has simple yaml file which includes Resource, Filter and Action

Resources: Custodian is able to target several cloud providers (AWS, GCP, Azure) and each provider has its own resource type.(eg ec2, s3 bucket)
Filters: Filters are the way in Custodian to target a specific subset of resources. It could be based on some date, tag etc. We can write our custom filter using the JMESPath expression.
Actions: Actions is the actual decision you make on resources that match the filter. This action can be as simple as sending a report to the owner, stating that the resource does not match the Cloud governance rule or delete the resource.

Both actions and filters can combine as many rules as you want to express your needs perfectly.

- name: first-policy
  resource: name-of-cloud-resource
  description: Description of policy
    filters:
      - (some filter that will select a subset of resource)
      - (more filters)
    actions:
      - (an action to trigger on filtered resource)
      - (more actions)

Cloud Custodian sample policy

Although Official docs cover most of the aws policies examples, We have picked up some policies which can be used from day 1 for cost saving and Compliance.

ebs-snapshots-month-old.yml

One of the most common issues the organization faces is the complexity of removing old ami,snapshot and volume which lie there in our environment for more than 1 years and add more bills. Eventually we have to write multiple custom scripts to deal with the situation.

Below is a simple policy which removes snapshots which are older than 30 days.

policies:
  - name: ebs-snapshots-month-old
    resource: ebs-snapshot
    filters:
      - type: age
        days: 30
        op: ge
    actions:
      - delete

Here is an example of how we can run the Cloud Custodian policy.

custodian run -v -s /tmp/output /tmp/ebs-snapshots-month-old.yml

Every time we run the Custodian command it creates/appends files inside policies.name output directory passed with -s option (eg. /tmp/output/ebs-snapshot-month-old/custodian-run.log)

custodian-run.log : All console logs are stored here
resources.json : Filtered resources list
metadata.json : Metadata about filtered resources
action-* : resources list on which action was taken
$HOME/.cache/cloud-custodian.cache : All cloud api call results are cached here. Default value is 15 minutes.

To get a filtered resource report we can run the below command. By default it provides reports in csv format but we can change it by passing --format json.

custodian report -s /tmp/output/ --format csv ebs-snapshots-month-old.yml

only-approved-ami.yml

Stop running ec2 which does not match with the trusted AMI list.

policies:
- name: only-approved-ami
  resource: ec2
  comment: |
    Stop running EC2 instances that are using invalid AMIs
  filters:
    - "State.Name": running
    - type: value
      key: ImageId
      op: not-in
      value:
          - ami-04db49c0fb2215364   # Amazon Linux 2 AMI (HVM)
          - ami-06a0b4e3b7eb7a300  # Red Hat Enterprise Linux 8 (HVM)
          - ami-0b3acf3edf2397475    # SUSE Linux Enterprise Server 15 SP2 (HVM)
          - ami-0c1a7f89451184c8b   # Ubuntu Server 20.04 LTS (HVM)
  actions:
    - stop

Security-group-check.yml

One of the more common issues that we see when Developers tend to allow all traffic on SSH while creating POC VM OR during testing we sometimes allow port 22 to ALL but forget to remove the rule. Below policy can take care of these issues by automatically removing SSH access from ALL and adding only VPN IP to the security group.

policies:
  - name: sg-remove-permission
    resource: security-group
    filters:
       - or:
             - type: ingress
               IpProtocol: "-1"
               Ports: [22]
               Cidr: "0.0.0.0/0"
             - type: ingress
               IpProtocol: "-1"
               Ports: [22]
               CidrV6: "::/0"
    actions:
      - type: set-permissions
        remove-ingress: matched
        add-ingress:
          - IpPermissions:
            - IpProtocol: TCP
              FromPort: 22
              ToPort: 22
              IpRanges:
                - Description: VPN1 Access
                  CidrIp: "10.10.0.0/16"

Support Kubernetes resources

We can now manage Kubernetes resources like deployment, pod, Daemonset, Volume. Below are some sample policies that we can write with Cloud Custodian.

Delete POC and untagged resources
Update labels and patch on k8 resources
Call webhooks based on findings

kubernetes-delete-poc-resource.yml

policies:
  - name: delete-poc-namespace
    resource: k8s.namespace
    filters:
    - type: value
      key: 'metadata.name'
      op: regex
      value: '^.*poc.*$'
    actions:
      - delete

  - name: delete-poc-deployments
    resource: k8s.deployment
    filters:
    - type: value
      key: 'metadata.name'
      op: regex
      value: '^.*poc.*$'
    actions:
      - delete

Note: Cloud Custodian kubernetes resources still work in progress. We can check the status of the plugin here.

What are the types of modes that we can call Cloud Custodian?

pull - Default method can be run manually. Preferred to add it in CICD tool cron.
periodic - Provision cloud resource (eg. Aws lambda with CloudWatch cron) as per policy and executes as scheduled.
Custom mode as per cloud provider - Executes when the event matches

Integrate Cloud Custodian with Jenkins CI

For simplicity we are using Cloud Custodian docker image and injecting the credentials as environment variables.

Note: secret file should have keys in upper case and default region. In case of kubernetes the KUBECONFIG file should be mounted inside the container.

export AWS_ACCESS_KEY_ID=<YOUR_AWS_ACCESS_KEY>
export AWS_SECRET_ACCESS_KEY=<YOUR_AWS_SECRET_ACCESS_KEY>
export AWS_DEFAULT_REGION=<YOUR_DEFAULT_REGION>

pipeline{
    agent{ label 'worker1'}
    stages{
        stage('cloudcustodian-non-prod'){
            steps{
                dir("non-prod"){
                    withCredentials([file(credentialsId: 'secretfile', variable: 'var_secretfile')])
                    {
                    sh '''
                    source $var_secretfile  > /dev/null 2>&1
                    env | grep "^AWS\\|^AZURE\\|^GOOGLE\\|^KUBECONFIG" > envfile

                    for files in $(ls | egrep '.yml|.yaml')
                    do
                        docker run --rm -t \
                        -v $(pwd)/output:/opt/custodian/output \
                        -v $(pwd):/opt/custodian/ \
                        --env-file envfile \
                        cloudcustodian/c7n run -v  -s /opt/custodian/output /opt/custodian/$files
                    done
                    '''
                    }
                }
            }
        }
        stage("cloudcustodian-prod"){
            steps{
                dir("prod"){
                    withCredentials([file(credentialsId: 'secretfile', variable: 'var_secretfile')])
                    {
                    sh '''
                    source $var_secretfile  > /dev/null 2>&1
                    env | grep "^AWS\\|^AZURE\\|^GOOGLE\\|^KUBECONFIG" > envfile

                    for files in $(ls | egrep '.yml|.yaml')
                    do
                        docker run --rm -t \
                        -v $(pwd)/output:/opt/custodian/output \
                        -v $(pwd):/opt/custodian/ \
                        --env-file envfile \
                        cloudcustodian/c7n run -v -s /opt/custodian/output /opt/custodian/$files
                    done
                    '''
                    }
                }
            }
        }
    }
}

Jenkins console output:

Tools and Features

Cloud Custodian has a number of add-on tools that have been developed by the community.

Multi Region and Multi Account support

We can use c7n-org plugging to configure multiple AWS, AZURE, GCP accounts and run them in parallel. Flag --region all can be used to run the same policy across all regions.

Notification

c7n-mailer plugin provides lots of flexibility for alert notifications. We can use webhook, email, queue service, Datadog, Slack and Splunk for alerts.

Auto-resource-tagging

c7n_trailcreator script will process cloudtrail records to create a sqlite db of resources and their creators, and then use that sqlitedb to tag the resources with their creator's name.

Logging and Reporting

It provides reporting in JSON and CSV format. We can also collect these metrics inside Cloud native logging and generate nice dashboards. We can store the logs locally, S3 or on Cloudwatch. A consistent logging format makes it easy to troubleshoot policies.

Custodian Dry run

In Dry run(--dryrun), the action part of policy is ignored. It shows what resources will be impacted by the policy. It is always best practice to do a dry run first before running the actual code.

Custodian Cache

When we execute any policy it fetches data from the cloud and stored locally for 15 min. Cache is used to minimize api calls. We can set the cache with --cache-period 0 option.

Editor integration

It can be integrated with Visual Studio Code for auto compilation and suggestion.

Custodian schema

We can use Custodian schema command to find out the type of resource, action and filters that are available inside Cloud Custodian.


custodian schema    #Shows all resource available in custodian
custodian schema aws    #Shows aws resource available in custodian
custodian schema aws.ec2     #Shows aws ec2 action and filters
custodian schema aws.ec2.actions     #Shows aws ec2 actions only
custodian schema aws.ec2.actions.stop    #Shows ec2 stop sample policy and schema

How is Cloud Custodian better than other tools?

Simplicity and Consistency of writing policies across multiple cloud platforms and kubernetes.
Multi account and Multi region support using c7n-org.
Support a wide range of Notification channels using c7n-mailer
Custodian's terraform provider enables writing and evaluating Custodian policies against Terraform IaaC modules.
Custodian has deep integration with AWS config. It can deploy any config-rule that is supported by config. Also It can automatically provision aws lambda for AWS custom config policy.
We can implement our custom policies in Python if you need to as it supports all rules as per Cloud providers SDK.
Cloud Custodian is an opensource CNCF Sandbox project.

Cloud Custodian Limitations

No Default Dashboard (Supports AWS native dashboard but We can also send metrics output to Elasticsearch/Grafana, etc. and create dashboard).
Cloud Custodian can not prevent custom layer validation pre deployments. It can only run periodically or based on some events.
Cloud Custodian does not have any in-built policies. We need to write all policies by ourselves. However it has a lot of good example policies(aws, azure, gcp) that we can use as reference.

Conclusion

Cloud Custodian enables us to define rules and remediation as one policy to facilitate a well-managed cloud infrastructure. We can also use it to write policies for managing Kubernetes resources like deployment, pod, etc. Compared to other cloud based governance tools It provides a very simple DSL to write policies and It’s Consistency across Cloud platforms. Custodian reduces the friction for innovating securely in the Cloud and also increases efficiency.

We can use Cloud Custodian to optimize our Cloud cost by implementing offhour and cleanup policies. It also includes lots of plugins like Multi account/region support, Wide range of Notification tools(Slack, SMTP, sqs, Datadog, Webhooks, etc), etc. We can find a list of Cloud Custodian plugins here.

That’s a wrap folks :) Hope the article was informative and you enjoyed reading it. I’d love to hear your thoughts and experience - let’s connect and start a conversation on LinkedIn.