Forem: Sivamuthu Kumar

GitHub Actions Scheduled Workflows: Understanding the Limitations and Possible Solutions

Sivamuthu Kumar — Sat, 21 Jan 2023 01:00:26 +0000

GitHub Actions is a powerful tool that allows developers to automate their workflow, but when it comes to scheduling workflows, it may not always work as expected. Many users have reported that their scheduled workflows are not triggering at the scheduled time, and in some cases, the delay can be as long as an hour. In this blog post, we will explore the limitations of GitHub Actions scheduled workflows, the reasons for these delays, and what can be done to ensure that your workflows are executed on time. We will also discuss various alternative solutions that can be used to trigger the workflows manually, guaranteeing the execution of your production tasks.

Scenario: Actions not running at the scheduled time.

I've recently written a GitHub action that generates reports by running scripts and sending an email of the report every Tuesday at 8 AM EST.

E.g.,

name: Generate Report

on:
  schedule:
    - cron: "0 13 * * 2" # UTC

jobs:
  generate:
    runs-on: ubuntu-latest
    steps:
    # Reduced the code for brevity
    - name: Generate report and Send email
      run: |
        generate-report.sh 
        send-email.sh

It took less than a minute to generate a report. I was expecting the email to be received at least by 8.05 AM. But it's not started. It took 30min to 1 hour to get the new workflow run queued.

One of the main reasons for delays in scheduled workflows is the fact that GitHub runs workflows on shared runners. There is no guarantee that the workflow will run at the exact scheduled time. When you set up a workflow schedule, you request GitHub to schedule it. However, many factors can affect the execution of the workflow, such as system load, queue size, and other workflows competing for resources.

This means that the resources available to run workflows are not dedicated to a specific user or repository and are subject to the demands of other users and workflows. As a result, there may be delays in the execution of a scheduled workflow, as it may need to wait for resources to become available.

Solution: Use external schedulers & Workflow Dispatch

One possible solution to overcome these limitations is manually triggering the workflow using an external scheduler. GitHub Actions supports the workflow_dispatch trigger, allowing you to trigger a workflow manually. This means you can use a third-party cron scheduling service, like IFTTT, Cronhub, Cronitor, etc., to request the GitHub API to trigger the workflow. By doing so, you can ensure that the workflow is executed at the desired time, regardless of the state of the shared runners or the cron schedule.

Add the workflow_dispatch to trigger the GitHub Actions manually.

name: Generate Report

on:
 workflow_dispatch:

jobs:
  generate:
    runs-on: ubuntu-latest
    steps:
    # Reduced the code for brevity
    - name: Generate report and Send email
      run: |
        generate-report.sh 
        send-email.sh

You could trigger the workflow using an HTTP call, for e.g,

curl -X POST -H "Accept: application/vnd.github+json" -H "Authorization: token YOUR_ACCESS_TOKEN" https://api.github.com/repos/OWNER/REPO/actions/workflows/WORKFLOW_ID/dispatches

This way, your workflow will be executed at the exact scheduled time, without any delays, regardless of the state of shared runners.

Conclusion

In summary, while GitHub Actions is a powerful tool for automating your workflow, its scheduled workflows may not always run at the exact scheduled time. This can be due to the shared runner's nature, where the system load, queue size and other workflows can affect the execution of your workflow. To ensure that your production tasks are executed on time, it is recommended to use an external scheduler to manually trigger the workflow. This way, you can guarantee the execution of your workflow and avoid delays.

I'm Siva - Director, DevOps & Principal Architect at Computer Enterprises Inc from Orlando. I'm an AWS Community builder. I write blogs and tutorials about Cloud, Containers, IoT, and DevOps. If you are interested, please follow me @ ksivamuthuon Twitter or check out my blogs at sivamuthukumar.com

Azure Terrafy: Import and Manage Existing Azure Resources with Terraform

Sivamuthu Kumar — Fri, 30 Dec 2022 12:06:35 +0000

As an Azure user, you may be looking for a way to manage your infrastructure with the power of Terraform. However, getting started with Terraform can be challenging, especially if you have existing resources that you want to include in your configuration. That's where Azure Terrafy comes in. Azure Terrafy is a tool that makes it easy to import your existing Azure resources into Terraform modules. In this blog post, I will introduce you to Azure Terrafy and show you how it can streamline your Azure resource management process. By the end, you will have a better understanding of how Terrafy can help you use Terraform to manage your infrastructure with ease.

Azure Terrafy is a tool that makes it easy to import your existing Azure resources into Terraform modules. Suppose you're an Azure user looking to manage your infrastructure with the power of Terraform. In that case, Azure Terrafy can save you time and effort by automating the process of incorporating your existing resources into your Terraform configuration. This is especially useful for those who have a "brownfield" environment, where their infrastructure already has a number of existing resources that need to be brought under the management of Terraform. It can save you a lot of time and effort. Without Terrafy, you would need to manually create a Terraform configuration file for each resource you want to manage. This can be tedious and error-prone, especially if you have many resources.

In addition to saving time and effort, using Azure Terrafy can also help you ensure that your Terraform configuration is accurate and up-to-date. By importing your existing resources into Terraform, you can ensure that your configuration reflects the current state of your resources rather than relying on potentially outdated documentation or manual configuration. This can help you avoid errors and ensure your infrastructure is managed effectively.

https://github.com/Azure/aztfy

Features and benefits of using Terrafy:

Automates the process of importing existing Azure resources into Terraform modules
Saves time and effort compared to manual resource management
Helps ensure that your Terraform configuration is accurate and up-to-date
Works with both Azure Resource Manager and Azure Classic resources
Can be easily integrated into existing Terraform workflows

How Azure Terrafy fits into the overall Azure and Terraform landscape:

Azure Terrafy is a tool specifically designed to help manage Azure resources in Terraform
It works alongside Azure Resource Manager to discover and import resources into Terraform modules
Azure Terrafy is just one piece of the puzzle regarding managing Azure infrastructure with Terraform. Other tools and technologies, such as Azure DevOps and Azure Functions, can also be used in conjunction with Terrafy to create a comprehensive infrastructure management solution.

Prerequisites

An Azure account and subscription
Terraform installed on your machine
The Azure CLI installed on your machine
A configured Azure CLI profile with the necessary permissions to read resources in your Azure subscription
Install aztfy from GitHub Releases - pre-compiled binaries on macOS, Windows & Linux platforms

Importing Existing Resources

Run the aztfy command to create a new Terraform configuration file on a single resource, resource group or query for the list of resources. You can pass the non-interactive option if you want to run in non-interactive mode. Here dapr-talk is the resource group name.
Review the generated Terraform configuration file to ensure that it accurately reflects the state of your Azure resources
Run the terraform plan command to preview the changes that will be made to your resources when you apply the Terraform configuration
If the plan looks correct, run the terraform apply the command to apply the Terraform configuration and bring your Azure resources under the management of Terraform.

Here are a few tips and best practices to keep in mind when using Terrafy to import your existing resources into Terraform:

Be sure to review the generated Terraform configuration file carefully before applying it. This will help you ensure that the configuration accurately reflects the state of your resources and avoid any errors or unintended consequences.
If you have many resources, it may take some time for Azure Terrafy to scan and generate the configuration file. Be patient and allow the process to complete before applying the configuration.
If you have resources that you don't want to include in your Terraform configuration, you can use the aztfy res or aztfy query command to import specific resources individually. This can be helpful if you only want to manage a subset of your resources with Terraform.
You must create terraform resources manually if there are not-supported / skipped resources.
Don't forget to run the terraform plan command before applying your configuration. This will help you identify potential issues or conflicts before they become a problem.

By following these steps and best practices, you can use Azure Terrafy to easily import your existing Azure resources into Terraform modules and start managing them with ease.

Limitations

Here are some limitations of Azure Terrafy to consider:

Only works with Azure resources: Terrafy cannot import resources from other cloud providers.
It may not be suitable for highly complex or customized infrastructure: Azure Terrafy does not support all resources. The resources which are not supported have to be done manually.
Limited context: The review is necessary to convert the sensitive configurations to terraform-managed variables.

Conclusion

In conclusion, I have found Azure Terrafy to be a valuable tool for managing Azure resources in Terraform. It has saved me time and effort compared to manual resource management, and its integration with Azure and Terraform has made it easy to use in various infrastructure management scenarios. However, like any tool, it has its limitations, such as only working with Azure resources and potentially complex for more customized infrastructure.

Overall, I recommend giving Azure Terrafy a try if you are looking for a tool to help manage your Azure resources with Terraform. Just be sure to consider its limitations and whether it fits the needs of your specific infrastructure. By following the steps outlined in this blog post and keeping in mind best practices like reviewing the generated code before applying it, you can use Azure Terrafy to manage your Azure resources with Terraform effectively.

I'm Siva - Director, DevOps & Principal Architect at Computer Enterprises Inc from Orlando. I'm an AWS Community builder. I write blogs and tutorials about Cloud, Containers, IoT, and DevOps. If you are interested, please follow me @ksivamuthu on Twitter or check out my blogs at sivamuthukumar.com

Boost Your Enterprise Security with GitHub Actions and the OSSF Score Card

Sivamuthu Kumar — Fri, 30 Dec 2022 12:02:37 +0000

In this blog post, we will cover how you can use the OSSF Scorecard to assess the security score of your repository and report the results in GitHub Advanced Security - Overview. This will allow you to assess the security scores of your repositories across your organization, providing a comprehensive view of your repositorys security posture.

The Open Source Security Foundation (OSSF) Scorecard is a tool that helps assess the security measures taken in a repository. It does this by performing a series of checks, each of which is assigned a score of 0-10. By using the OSSF Scorecard, you can assess the risks that dependencies introduce, understand specific areas where you can improve the security posture of your project, and make informed decisions about accepting those risks or evaluating alternative solutions.

https://github.com/ossf/scorecard

Some of the checks included in the OSSF Scorecard are:

Branch-Protection: This check assesses whether the project uses Branch Protection to ensure that code is reviewed before it is merged.
CI-Tests: This check looks for the presence of tests that are run in CI, such as GitHub Actions.
Code-Review: This check assesses whether the project requires code review before code is merged.
Dangerous-Workflow: This check looks for the use of dangerous coding patterns in GitHub Action workflows, which can pose a security risk.
Dependency-Update-Tool: This check looks for the use of tools to help update dependencies, which can help keep the project up-to-date and secure.
License: This check assesses whether the project has a declared license, which is important for ensuring legal compliance.
Maintained: This check looks for projects that are at least 90 days old and are actively maintained.
Pinned Dependencies: This check looks for the declaration and pinning of dependencies, which can help ensure that the project is using stable and secure versions of those dependencies.
Packaging: This check looks for the building and publishing of official packages from CI/CD, such as GitHub Publishing.
SAST: This check looks for the use of static code analysis tools such as CodeQL or other SAST tools such as SonarCloud, which can help identify vulnerabilities in the code.
Security Policy: This check looks for the presence of a security policy, which can help outline the steps taken to ensure the security of the project.
Signed-Releases: This check looks for the use of cryptographic signing for releases, which can help ensure the authenticity and integrity of those releases.
Token-Permissions: This check assesses whether the project declares GitHub workflow tokens as read-only, which can help prevent unauthorized access to sensitive information.
Vulnerabilities: This check looks for unfixed vulnerabilities in the project and uses the OSV service to assess the risk level.
Webhooks: This check looks for the presence of tokens in webhooks that are used to authenticate the origins of requests. This can help prevent unauthorized access to the project.

You can use GitHub Actions to automate the process of updating an OSSF scorecard for a project. To do this, you will need to set up a workflow that runs the OSSF Scorecard and then parses and displays the results.

In this tutorial, we will walk through the process of using GitHub Actions to automate the process of updating an OSSF scorecard for a project. We will cover the following steps:

Setting up a GitHub repository for your project.
Creating a workflow file to automate OSSF scorecard updates
Running security scans and updating dependencies with GitHub Actions
Viewing and updating your OSSF scorecard

Setting up a GitHub repository for your project

If you don't already have a GitHub repository, the first step is to create one. To do this, log in to your GitHub account and click the "New repository" button. Give your repository a name and description, and choose whether you want it to be public or private.

Creating a workflow file to automate OSSF Scorecard updates

Next, we will create a workflow file to define the steps performed by our GitHub Action.

You can set up this directly or navigate to the Security tab Setup Code scanning (or Add more scanning tools).
Choose the OSSF Scorecards supply-chain security analysis from the list and set up the workflow. It will create a workflow file like the one below.
Here is an example workflow file that runs a security scorecard scan and uploads the findings in Code scanning alerts whenever code is pushed to the main branch:

In this example, the on block defines the trigger for the workflow, which is a push event to the main branch. The jobs block defines the steps that are performed by the workflow. In this case, the workflow has an analysis job that runs on an Ubuntu virtual machine. The job has three steps:

actions/checkout@<sha>: This action checks out the code from the repository to the virtual machine.
ossf/scorecard-action@<sha>: This action runs an OSSF Scorecard section on the repo to identify any misconfigurations based on the OSSF rules.
actions/upload-artifact@<sha>:: This action runs an upload artifact - the sarif file result from the scorecard action
actions/upload-artifact@<sha>:: This action uploads OSSF Scorecard results to the code scanning alerts to the dashboard.

Code scanning Alerts

Using the OSSF Score card in conjunction with code scanning alerts can be a powerful way to ensure the security of your projects in the enterprise. By integrating the OSSF Scorecard with code scanning alerts, you can receive notifications whenever a check fails or a score falls below a certain threshold. This can help you stay informed about the security of your repository and take prompt action to fix any identified issues.

You can see the details of the check and the remediation steps to fix it.

To view code scanning alerts of all your repositories in the organization, you can go to the "Security" section. Monitoring code scanning alerts is an important part of maintaining the security of your repositories in the enterprise. Regularly reviewing and addressing any identified vulnerabilities can help ensure that your projects are secure and protect your organization against potential vulnerabilities. Here is the screenshot of the sample Security Risk of your organization.

Best Practices

It is important to note that the OSSF Scorecard is just one tool that can help you assess the security of your repositories. It is recommended that you use a combination of tools and best practices to ensure the security of your projects in the enterprise. Some other best practices to consider include the following:

Regularly updating dependencies and keeping them up-to-date.
Using secure coding practices and following best practices for secure development.
Regularly scanning your code for vulnerabilities using tools such as CodeQL or other SAST tools.
Implementing branch protection and code review processes to ensure that code is reviewed before it is merged.
Creating and maintaining a security policy that outlines the steps taken to ensure the security of the project.
Cryptographically signing releases to ensure the authenticity and integrity of those releases.

By following these best practices and using tools like the OSSF Scorecard, you can help improve the security of your projects in the enterprise and protect your organization against potential vulnerabilities.

Conclusion

Using the OSSF Scorecard, you can get a comprehensive view of the security measures taken in your repository and identify areas for improvement. You can also use the scorecard in conjunction with code scanning alerts to receive notifications when potential vulnerabilities or security issues are discovered, allowing you to take prompt action to fix those issues.

Overall, the OSSF Scorecard is a valuable tool for assessing and improving the security of your projects in the enterprise. By using it regularly, you can ensure that your projects are secure and well-maintained, and you can demonstrate to your customers and stakeholders that you are committed to maintaining a high level of security.

I'm Siva - Director, DevOps & Principal Architect at Computer Enterprises Inc from Orlando. I'm an AWS Community builder. I write blogs and tutorials about Cloud, Containers, IoT, and DevOps. If you are interested, please follow me @ksivamuthu on Twitter or check out my blogs at sivamuthukumar.com

EKS Observability Infrastructure as Code - AWS Observability Accelerator

Sivamuthu Kumar — Mon, 26 Dec 2022 00:24:13 +0000

Observability is an essential part of any cloud-native environment. When it comes to Amazon EKS clusters, observability is an even more critical factor. This blog post will explore how you can provision an observability landscape to monitor and manage your EKS clusters using AWS Observability Accelerator for Terraform.

The AWS Observability accelerator for Terraform is a powerful terraform module that enables developers to quickly and easily set up observability solutions for their Amazon Elastic Container Service for Kubernetes (EKS) clusters using Amazon Web Services (AWS) observability services. This tool includes a core module that helps you configure your cluster with the AWS Distro for OpenTelemetry (ADOT) Operator for EKS, Amazon Managed Service for Prometheus, and Amazon Managed Grafana.

In addition to the core module, the AWS Observability accelerator also includes a set of workload modules that provide curated ADOT collector configurations, Grafana dashboards, Prometheus recording rules, and alerts. These modules allow you to leverage the power of these tools to monitor and troubleshoot your applications and infrastructure in real-time.

%[https://github.com/aws-observability/terraform-aws-observability-accelerator]

In this post, we will set up the EKS cluster using terraform. Then we will provision AWS Managed service for Prometheus, AWS Managed service for Grafana, Install AWS Distro for Open Telemetry in Cluster to scrape the metrics and necessary recording rules, and dashboards for Prometheus and Grafana - all these in only a few lines of Infrastructure as Code.

Let’s start with the setup of initializing the EKS cluster using the EKS Blueprints terraform module.

Initialize the EKS cluster using the EKS Blueprints terraform module. EKS blueprints are modular constructs available in CDK and as terraform modules that can provision fully functional production-ready kubernetes cluster and modular add-ons such as service mesh, autoscalers, networking, storage, gitops, etc.

module "eks_blueprints" {
  source          = "github.com/aws-ia/terraform-aws-eks-blueprints?ref=v4.19.0"
  cluster_name    = var.cluster_name
  cluster_version = "1.20"

  vpc_id             = var.vpc_id
  private_subnet_ids = var.private_subnet_ids
  public_subnet_ids  = var.public_subnet_ids

  managed_node_groups = {
    t3_small = {
      node_group_name = "node-group-t3-small"
      instance_types  = ["t3.small"]
      subnet_ids      = var.private_subnet_ids
      min_size        = 1
      max_size        = 2
      desired_size    = 1
    }
  }

  tags = var.tags
}

You can then specify which observability services you want to enable, such as the ADOT operator, Amazon Managed Prometheus, and Amazon Managed Grafana.

module "eks_observability_accelerator" {
  source = "github.com/aws-observability/terraform-aws-observability-accelerator?ref=v1.5.0"

  aws_region     = var.aws_region
  eks_cluster_id = module.eks_blueprints.eks_cluster_id

    enable_amazon_eks_adot    = true
  enable_managed_prometheus = true
  enable_alertmanager       = true
    enable_managed_grafana    = true

  tags = var.tags
}

You can also specify your own instance IDs if you want to reuse existing managed services or disable the creation of managed services altogether.

module "eks_observability_accelerator" {
  source = "github.com/aws-observability/terraform-aws-observability-accelerator?ref=v1.5.0"

  aws_region     = var.aws_region
  eks_cluster_id = module.eks_blueprints.eks_cluster_id

    enable_amazon_eks_adot    = true
  enable_managed_prometheus = true
  enable_alertmanager       = true

    # Reuse existing setup by disabling the creation of managed grafana service
    # and pass the existing service workspace id
  enable_managed_grafana       = false
  managed_grafana_workspace_id = var.managed_grafana_workspace_id
  grafana_api_key              = var.grafana_api_key

  tags = var.tags
}

Set up the default Grafana dashboards, alerting rules, and metrics scraping configuration from AWS Observability Accelerator for Terraform modules. You can customize them as needed.

module "workloads_infra" {
  source = "github.com/aws-observability/terraform-aws-observability-accelerator/modules/workloads/infra"

  eks_cluster_id = module.eks_observability_accelerator.eks_cluster_id

  dashboards_folder_id            = module.eks_observability_accelerator.grafana_dashboards_folder_id
  managed_prometheus_workspace_id = module.eks_observability_accelerator.managed_prometheus_workspace_id

  managed_prometheus_workspace_endpoint = module.eks_observability_accelerator.managed_prometheus_workspace_endpoint
  managed_prometheus_workspace_region   = module.eks_observability_accelerator.managed_prometheus_workspace_region

  # optional, defaults to 60s interval and 15s timeout
  prometheus_config = {
    global_scrape_interval = "60s"
    global_scrape_timeout  = "15s"
  }

  depends_on = [
    module.eks_observability_accelerator
  ]
}

Once you have included the AWS Observability Accelerator in your Terraform configuration and specified which services you want to enable, you can use Terraform to provision the observability landscape for your EKS cluster.
```
terraform plan -var-file env/dev.tfvars
terraform apply -var-file env/dev.tfvars
```

The complete Observability Landscape has been set up for you - including ADOT (AWS Distro for Open Telemetry collectors), AWS Managed Services for Prometheus (Prometheus endpoints where ADOT writes the metrics), AWS Managed Services for Grafana (where it can query from Prometheus data sources to build dashboards and display alerts).

You can view the Prometheus Endpoint URL, alerting, and recording rules in the AWS console.

Navigate the Grafana Dashboard in the AWS Managed Services for the Grafana workspace, and configure the user to view/edit the dashboards. The necessary dashboards are already created for the user to view.

Kubernetes - Cluster
Kubernetes - Namespace(Workloads)
Kubernetes - Node (Pods)
Kubernetes - Workload
Kubernetes - Kubelet
Node Exporter - Nodes

You can navigate to the individual dashboard and view the charts based on the metrics scraped from the infrastructure and workloads.

Conclusion

In conclusion, the AWS Observability Accelerator for Terraform is a powerful terraform module that allows you to quickly and easily set up observability solutions for your EKS clusters. It includes a core module and a set of workload modules that provide curated configurations, dashboards, recording rules, and alerts to help you monitor and troubleshoot your applications and infrastructure in real-time. Using the AWS Observability Accelerator, you can leverage the power of tools like Prometheus and Grafana to visualize and analyze your metrics and logs, enabling you to optimize and improve the performance and reliability of your cloud-native environments.

I'm Siva - working as Director, DevOps & Principal Architect at Computer Enterprises Inc from Orlando. I'm an AWS Community builder. I will write a lot about Cloud, Containers, IoT, and DevOps. If you are interested, please follow me @@ksivamuthu on Twitter or check out my blogs at sivamuthukumar.com

AWS re:Invent 2022 Announcements - Data, Infrastructure & Security Highlights

Sivamuthu Kumar — Wed, 30 Nov 2022 14:46:50 +0000

Amazon made a couple of announcements today (November 29, 2022) at AWS re: Invent in Las Vegas, focusing on Data, Analytics, Infrastructure & Security. We will see the announcements that excite me in this blog post.

Data & Analytics:

Zero ETL future

When you build a data platform - ETL takes primary effort. AWS committed to building the Zero ETL future and made two great announcements toward that.

- [Amazon Aurora zero-ETL integration with Amazon Redshift](https://aws.amazon.com/about-aws/whats-new/2022/11/amazon-aurora-zero-etl-integration-redshift/)
    - enables near-real-time analytics and ML on transaction data.
    - consolidate data from multiple Aurora database
    - continuous updates
    - serverless - no infrastructure to manage
- Amazon Redshift Integration for Apache Spark
    - Now, you can run Spark queries on REshift data from EMR, Glue, and Sagemaker within seconds. No need to move the data

Amazon Datazone - ML based Data management service
- A data management service that helps orgs catalog, share and govern data.
- Integrates with Redshift, Athena and Quicksights and provides APIs to third party sources
- AWS Clean Room - Data sharing / masking services.
  1. I think AWS is running out the names for the product. ;-)
Amazon QuickSight - ML powered
- It includes AI-enhanced automated data preparation, making it fast & easy to enable data for natural language questions.
Amazon OpenSearch Serverless
- Run search and analytics workloads without managing clusters.
- It’s one of the features I expected - finally, it got announced.
Amazon RDS Blue/Green Deployments
- A new feature for Amazon Aurora with MySQL compatibility, Amazon RDS for MySQL, and Amazon RDS for MariaDB that enables you to make database updates safer, simpler, and faster.
- With just a few steps, you can use Blue/Green Deployments to create a separate, synchronized, fully managed staging environment that mirrors the production environment. The staging environment clones your production environment’s primary database and in-Region read replicas. Blue/Green Deployments keep these two environments in sync using logical replication

Infrastructure

Graviton: NEW 3rd generation processor. 25% faster & use 60% LESS energy.
1. Strong momentum for Graviton
2. 500x time increase: delivering highest performing #MachineLearning workloads. 70% lower cost per inference.
AWS announces EC2 Inf2 instances
AWS Simspace Weaver - Run large-scale spatial simulations in the cloud

Security

Amazon Security Lake: centralizes security data of your organization from AWS and third-party systems for quick risk response. Supports OCSF standard format. Amazon Security Lake is the first service/initiative to support OCSF open standard format - to normalize the structure for security findings across vendors and products.
AWS Guard Duty - Runtime vulnerability detection for containers., attempts to access host nodes, etc. This feature will be available soon and integrates with Amazon EKS

Industries

Amazon Omics - Genome large-scale multimodal analysis that integrates with Amazon Healthlake or Sagemaker
AWS Supply Chain: improves visibility into systems to lower costs & improve #CX - supply chain resiliency.

Summary

There were exciting announcements in today's keynote focused on Data, Analytics & Infrastructure mostly. I'm excited to about learning these new announcements and help enterprises on their successful journey - that innovates, analyzes, and scales their enterprise data analytics platform.

I'm Siva - working as Sr. Software Architect at Computer Enterprises Inc from Orlando. I'm an AWS Community builder. I will write a lot about Cloud, Containers, IoT, and Devops. If you are interested, please follow me @ksivamuthu on Twitter or check out my blogs at blog.sivamuthukumar.com!

EKS Blueprints - Modular Constructs for your Kubernetes cluster

Sivamuthu Kumar — Sun, 08 May 2022 03:13:47 +0000

Amazon Elastic Kubernetes Service (EKS) helps you manage, scale, and deploy your Kubernetes clusters. In the enterprise cluster setup, you need more than the cluster setup. To bootstrap, the kubernetes cluster requires significant features, including automated node management, ingress controller, monitoring, logging, security, networking tools, etc.

With my experience with other cloud provider platform Kubernetes offerings such as GKE & AKS, EKS was behind in the management features - logging, ingress controllers, etc. We need to set it up manually to get the basic features. That was hard to get into the Kubernetes platform - especially if it’s the first time onboarding to Kubernetes experience and/or EKS cluster. Later the Kubernetes components from AWS speed up the operation’s onboarding process easier.

AWS EKS Blueprints - AWS is making the operations and developer experience greater by providing modular Infrastructure as Code modules to expedite the process of onboarding the cluster and open source tools and security controls with production readiness.

What are AWS EKS Blueprints?

EKS Blueprints is a collection of Infrastructure as Code (IaC) modules that will help you configure and deploy EKS clusters across accounts and regions. You can use EKS Blueprints to easily bootstrap an EKS cluster with Amazon EKS add-ons as well as a wide range of popular open-source add-ons, including Prometheus, Karpenter, Nginx, Traefik, AWS Load Balancer Controller, Fluent Bit, Keda, Argo CD, and more. EKS Blueprints also helps you implement relevant security controls needed to operate workloads from multiple teams in the same cluster.

Highlights of the features:

Deploy Well-Architected EKS clusters across accounts and regions
Manage cluster configuration, including add-ons from a single GIt repo.
Teams and Access management
Continuous Delivery pipelines for deploying infrastructure
GitOps - based workflows for workloads

Courtesy: Image from AWS Blog.

Imperative / Declarative

When it comes to Infrastructure as Code, there are two options - Declarative (Terraform, yamls, etc.) and Imperative (CDK, etc.). AWS EKS Blueprints are available in both declarative and imperative options.

AWS EKS Blueprints for Terraform

In AWS EKS Blueprints for Terraform - the tools are available as terraform modules to implement. You can bootstrap the kubernetes cluster and use the adds-on modules to install the open-source tools.

aws-ia / terraform-aws-eks-blueprints

Configure and deploy complete EKS clusters.

AWS EKS Blueprints for CDK

In AWS EKS Blueprints for CDK - the tools are available as CDK constructs to implement. You can bootstrap the kubernetes cluster and use the adds-on modules to install the open-source tools.

aws-quickstart / cdk-eks-blueprints

AWS Quick Start Team

Demo Walkthrough

In today’s blog, we are going to set up the EKS Blueprint CDK and set up the necessary add-ons and the team structure.

Install the AWS-CDK
```
npm install -g aws-cdk
```

Initialize the CDK application

mkdir eks-blueprint
cdk init app --language typescript

Create the EKS Blueprint Construct and call it from bin/.ts

## ./bin/eks-blueprint.ts
#!/usr/bin/env node
import 'source-map-support/register';
import * as cdk from 'aws-cdk-lib';
import { EKSBlueprintConstruct } from '../lib/eks-blueprint-stack';

const app = new cdk.App();
new EKSBlueprintConstruct().build(app, 'siva-dev');

The EKS Blueprint Construct has three parts.

Create a generic EKS cluster with a node group configuration
Add the addons
Add the teams - dev team and platform team.


export class EKSBlueprintConstruct {

  build(scope: Construct, id: string, props?: StackProps) {
    const devTeam = new ApplicationTeam({ name: 'dev', users: this.getUserArns(scope, 'devUsers') });
    const platformTeam = new PlatformTeam({ name: 'platform', users: this.getUserArns(scope, 'platformUsers') });

    const teams = [devTeam, platformTeam];

    const addOns = [
      new VpcCniAddOn(),
      new CoreDnsAddOn(),
      new KubeProxyAddOn(),
      new MetricsServerAddOn(),
      new AwsLoadBalancerControllerAddOn(),
      new ContainerInsightsAddOn(),
      new KarpenterAddOn()
    ];

    const clusterProvider = new GenericClusterProvider({
      version: KubernetesVersion.V1_21,
      managedNodeGroups: [{
        id: `${id}-nodegroup`,
        instanceTypes: [new InstanceType('t3.small')],
        minSize: 1,
        maxSize: 2
      }],
    })

    EksBlueprint.builder()
      .addOns(...addOns)
      .clusterProvider(clusterProvider)
      .teams(...teams)
      .region('us-east-1')
      .build(scope, id, props);
  }
    ...
}

You can add the context properties that have devUsers and platformUsers that has a list of user/role ARN to create the team.

"context": {
    "devUsers": [
      "arn:aws:iam::xxxxxxxxxxx:user/user_name_1",
      "arn:aws:iam::xxxxxxxxxxx:user/user_name_2"
    ],
    "platformUsers": [
           "arn:aws:iam::xxxxxxxxxxx:user/admin_name"
        ]
  }

```tsx
  private getUserArns(scope: Construct, key: string): ArnPrincipal[] {
    const context: string[] = scope.node.tryGetContext(key);
    if (context && context.length > 0) {
      return context.map(e => new ArnPrincipal(e));
    }
    return [];
  }
```

You will see the CDK output has kubernetes config commands, the role for the dev and platform team.

siva-dev: creating CloudFormation changeset...

 ✅  siva-dev

✨  Deployment time: 1437.28s

Outputs:
siva-dev.devsa = dev-sa
siva-dev.devteamrole = arn:aws:iam::xxxxxxxxxx:role/dev-role-name
siva-dev.platformteamadmin = none
siva-dev.sivadevClusterName9C1D1E82 = siva-dev
siva-dev.sivadevConfigCommandF9D1C39C = aws eks update-kubeconfig --name siva-dev --region us-east-1 --role-arn <role_name>
siva-dev.sivadevGetTokenCommandA8918546 = aws eks get-token --cluster-name siva-dev --region us-east-1 --role-arn <role_name>

Caveats

The latest kubernetes version is not supported yet. Having issues with k8s version 1.22. You may see issues. Please use v1.21 at this time. https://github.com/aws-quickstart/cdk-eks-blueprints/issues/350

Also, you’ve to set the region - for the AWS Load balancer controller addon to work as the container images are derived from the region variable.

Helm release - testing - It will be a great addition if the CDK is able to fail if the addons fail to install. For e.g., without setting the region, the AWS Load balancer controller was in a pending state.

Conclusion

AWS EKS Blueprints is definitely a big step in making the dev and operation team's life easier. AWS is also working with Industrial partners and open source tools such as Datadog, Kasten.io, ArgoCD, Hashicorp, Snyk, etc.

In the next post of this series, we will see how to set up the CI/CD of the infrastructure and application workloads using EKS blueprints. And also how to extend the EKS blueprint to bring your own set of tools for your enterprise.

I'm Siva - working as Sr. Software Architect at Computer Enterprises Inc from Orlando. I'm an AWS Community builder and Auth0 Ambassador. I am going to write a lot about Cloud, Containers, IoT, and Devops. If you are interested in any of that, make sure to follow me if you haven’t already. Please follow me @ksivamuthu on Twitter or check out my blogs at blog.sivamuthukumar.com!

AWS Copilot GitHub Actions

Sivamuthu Kumar — Mon, 02 May 2022 18:39:14 +0000

In this blog, I’m going to walk through how to set up AWS Copilot in GitHub actions.

AWS Copilot

AWS Copilot is an open-source command-line interface that makes it easy for developers to build, release, and operate production-ready containerized applications on AWS App Runner, Amazon ECS, and AWS Fargate.

Initialize the Copilot Application

Using copilot, you can initialize the entire infrastructure to run your containerized apps. Let’s take a look at an example. In this repository, we’ve demo nginx container that serves the index.html file. You can check out this repo here.

Initialize the application using the copilot command-line tool.

 copilot init --app demo \
        --name app \
      --type 'Load Balanced Web Service'\
      --dockerfile './Dockerfile

Copilot will create the necessary infrastructure needed for your app in the AWS account. It will create an app service in the demo app, exposed to the internet using a load balancer. The container for running services is mentioned using dockerfile. The CLI will build the container from the docker file, set up ECR, push the image to ECR and configure the ECS service to pull the image from ECR.

Ok great, we'll set up a Load Balanced Web Service named app in application demo listening on port 80.

✔ Created the infrastructure to manage services and jobs under application demo.

✔ The directory copilot will hold service manifests for application demo.

✔ Wrote the manifest for service app at copilot/app/manifest.yml
Your manifest contains configurations like your container size and port (:80).

✔ Created ECR repositories for service app.

All right, you're all set for local development.
Deploy: Yes

✔ Linked account 495775103319 and region us-east-1 to application demo.

✔ Proposing infrastructure changes for the demo-test environment.
- Creating the infrastructure for the demo-test environment.             [create complete]  [82.9s]
  - An IAM Role for AWS CloudFormation to manage resources               [create complete]  [15.9s]
  - An ECS cluster to group your services                                [create complete]  [9.4s]
  - An IAM Role to describe resources in your environment                [create complete]  [16.7s]
  - A security group to allow your containers to talk to each other      [create complete]  [5.7s]
  - An Internet Gateway to connect to the public internet                [create complete]  [21.7s]
  - Private subnet 1 for resources with no internet access               [create complete]  [5.7s]
  - Private subnet 2 for resources with no internet access               [create complete]  [5.7s]
  - Public subnet 1 for resources that can access the internet           [create complete]  [12.6s]
  - Public subnet 2 for resources that can access the internet           [create complete]  [9.3s]
  - A Virtual Private Cloud to control networking of your AWS resources  [create complete]  [18.2s]
✔ Created environment test in region us-east-1 under application demo.

Copilot Pipeline

Copilot has a pipeline command to set up the Automated release pipeline using AWS CodePipeline for your application. You can learn more about it here.

Copilot in GitHub Actions

This article is going to focus on how to deploy the application using Copilot using GitHub Actions. Let’s walk through the setup step by step and then I wrote a GitHub action to set up for you with easy configuration.

Installing Copilot CLI

The AWS copilot CLI has to be downloaded and installed to access. The AWS Copilot binaries are released on GitHub. We can download the platform-specific binaries for a specific version or the latest version from the GitHub releases pages here.

Releases · aws/copilot-cli

Setting up OIDC provider

OpenID Connect (OIDC) allows your GitHub Actions workflows to access resources in Amazon Web Services (AWS), without needing to store the AWS credentials as long-lived GitHub secrets.

We can configure AWS to trust GitHub's OIDC as a federated identity and includes a workflow for the [aws-actions/configure-aws-credentials](https://github.com/aws-actions/configure-aws-credentials) that use tokens to authenticate to AWS and access resources.

Copilot CLI Usage

You can use the copilot after installing the tool in your PATH.

copilot --version

You can use the deploy command using copilot cli

copilot deploy --name demo --env test

I’ve written a GitHub Actions wrapping these functionalities

Installing Copilot CLI in Tool path
Deploy the Copilot Application

AWS GitHub Copilot Actions

ksivamuthu / aws-copilot-github-action

AWS Copilot GitHub Action

This repo contains the github actions for installing AWS Copilot cli and deploying app. The AWS Copilot CLI is a tool for developers to build, release and operate production ready containerized applications on AWS App Runner, Amazon ECS, and AWS Fargate.

Usage

To install copilot-cli in your github actions.

  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v1
        with:
          role-to-assume: arn:aws:iam::111111111111:role/my-github-actions-role-test
          aws-region: us-east-1
      - uses: ksivamuthu/aws-copilot-github-action@v0.0.1
        with:
          command: install
      - run: |
          copilot --version

To deploy the app

  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v1
        with:
          role-to-assume: arn:aws:iam::111111111111:role/my-github-actions-role-test
          aws-region: us-east-1
      - uses: ksivamuthu/aws-copilot-github-action@v0.0.1
        with:
          command: deploy
          app: your-awesome-app
          env: prod
          force: false # optional

View on GitHub

AWS Copilot GitHub Actions Usage

To install the copilot cli in the path. You can add the ksivamuth/aws-copilot-github-action@v0.0.1 steps with commands install to add the copilot cli in tool path. You can access the copilot cli - for e.g we are checking the version here.

  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v1
        with:
          role-to-assume: arn:aws:iam::111111111111:role/my-github-actions-role-test
          aws-region: us-east-1
      - uses: ksivamuthu/aws-copilot-github-action@v0.0.1
        with:
          command: install
      - run: |
          copilot --version

To deploy the application using copilot. Pass the application name and environment.


  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v1
        with:
          role-to-assume: arn:aws:iam::111111111111:role/my-github-actions-role-test
          aws-region: us-east-1
      - uses: ksivamuthu/aws-copilot-github-action@v0.0.1
        with:
          command: deploy
          app: your-awesome-app
          env: prod
          force: false # optional

The above example is in the demo repository here.

When the changes are pushed into GitHub, the action build the docker image and push the image repository into ECR. Then the pushed docker image is deployed into configured environment for this ECS or AppRunner application.

Once the workflow is succeed, you can see the latest version of the application deployed.

Conclusion

🚀 AWS Copilot supercharges your application - one cli tool to set up infrastructure, build your application with many services, setting up the pipeline to automate release, monitor the status of stack and application, and with add-ons. In this blog, we took a look how to integrate the Copilot app using GitHub Actions.

I'm Siva - working as Sr. Software Architect at Computer Enterprises Inc from Orlando. I'm an AWS Community builder, Auth0 Ambassador and I am going to write a lot about Cloud, Containers, IoT, and Devops. If you are interested in any of that, make sure to follow me if you haven’t already. Please follow me @ksivamuthu Twitter or check out my blogs at blog.sivamuthukumar.com!

Auth0 JWT Middleware in Go - Gin Web Framework

Sivamuthu Kumar — Sat, 29 Jan 2022 06:59:17 +0000

In today's blog post, we will see how to validate the JWTs using Auth0 Golang JWT middleware using Gin Web Framework. Gin is a web framework written in Go (Golang). In deep-dive, we will see how to integrate the Auth0 Golang JWT middleware to verify JWTs generated using both HS256 and RS256 using secret and JWKs.

Auth0 Golang JWT middleware stable version v2.0.0 released on Jan 19, 2022

Release v2.0.0 · auth0/go-jwt-middleware

Gin Web Framework

Gin is a high-performance micro-framework that can build web applications and microservices. It makes it simple to build a request handling pipeline from modular, reusable pieces. It does this by allowing you to write middleware that can be plugged into one or more request handlers or groups of request handlers.

Let's create a simple API using Golang Gin Web Framework.

Install Gin package

  go get -u github.com/gin-gonic/gin

Import it in your code and run simple api

  package main

  import (
    "context"
    "net/http"
    "github.com/gin-gonic/gin"
  )

  type Product struct {
    ID    int     `json:"id"`
    Title string  `json:"title"`
    Code  string  `json:"code"`
    Price float32 `json:"price"`
  }

  func main() {
    r := gin.Default()

    r.GET("/products", func(c *gin.Context) {
        products := []Product{
            {ID: 1, Title: "Product 1", Code: "p1", Price: 100.0},
            {ID: 2, Title: "Product 2", Code: "p2", Price: 200.0},
            {ID: 3, Title: "Product 3", Code: "p3", Price: 300.0},
        }
        c.JSON(http.StatusOK, products)
    })

    // Listen and Server in 0.0.0.0:5000
    r.Run(":5000")
  }

Add JWT Middleware

Before configuring the Auth0 APIs, lets' integrate the Auth0 Golang JWT middleware.

GitHub - auth0/go-jwt-middleware: A Middleware for Go Programming Language to check for JWTs on HTTP requests

go get github.com/auth0/go-jwt-middleware/v2

Auth0 Golang JWT middleware is the HTTP middleware handler. To use it in the Gin Web framework, we need a wrapper to wrap the common HTTP middleware to the Gin middleware handler. For that purpose, I'm using Gin Adapter from Gareth Watts. https://github.com/gwatts/gin-adapter

package main

import (
    "context"
    "net/http"

    jwtmiddleware "github.com/auth0/go-jwt-middleware/v2"
    "github.com/auth0/go-jwt-middleware/v2/validator"
    "github.com/gin-gonic/gin"
    "github.com/gwatts/gin-adapter"
)

type Product struct {
    ID    int     `json:"id"`
    Title string  `json:"title"`
    Code  string  `json:"code"`
    Price float32 `json:"price"`
}

func main() {
    r := gin.Default()

    keyFunc := func(ctx context.Context) (interface{}, error) {
        return []byte("secret"), nil
    }

    jwtValidator, _ := validator.New(keyFunc, validator.HS256, "http://localhost:5000", []string{"api:read"})
    jwtMiddleware := jwtmiddleware.New(jwtValidator.ValidateToken)

    // Wrap the http handler with gin adapter
    r.Use(adapter.Wrap(jwtMiddleware.CheckJWT))

    r.GET("/products", func(c *gin.Context) {
        products := []Product{
            {ID: 1, Title: "Product 1", Code: "p1", Price: 100.0},
            {ID: 2, Title: "Product 2", Code: "p2", Price: 200.0},
            {ID: 3, Title: "Product 3", Code: "p3", Price: 300.0},
        }
        c.JSON(http.StatusOK, products)
    })

    // Listen and Server in 0.0.0.0:5000
    r.Run(":5000")
}

Let's test this API with JWT URLs.

curl --location --request GET 'http://localhost:5000/products' \
--header 'Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyLCJpc3MiOiJodHRwOi8vbG9jYWxob3N0OjUwMDAiLCJhdWQiOiJhcGk6cmVhZCJ9.hbgwKW_RILeXXDDUv5bVK3WgjtvqoK5IiuisgnFWefY'

You can generate a new JWT with issuer [http://localhost:5000](http://localhost:5000) and audience api:read with the HS256 algorithm with a shared secret secret

➜  ~  curl --location --request GET 'http://localhost:5000/products' \
--header 'Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyLCJpc3MiOiJodHRwOi8vbG9jYWxob3N0OjUwMDAiLCJhdWQiOiJhcGk6cmVhZCJ9.hbgwKW_RILeXXDDUv5bVK3WgjtvqoK5IiuisgnFWefY' | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   160  100   160    0     0   6011      0 --:--:-- --:--:-- --:--:-- 20000
[
  {
    "id": 1,
    "title": "Product 1",
    "code": "p1",
    "price": 100
  },
  {
    "id": 2,
    "title": "Product 2",
    "code": "p2",
    "price": 200
  },
  {
    "id": 3,
    "title": "Product 3",
    "code": "p3",
    "price": 300
  }
]

Configure Auth0 APIs

Now it's time to add authorization to the API using Auth0. Let's configure Auth0 APIs with the signing algorithm RS256 and change the Auth0 Golang middleware code to support RS256.

Add the permission read:products in the API

Change the Auth0 middleware to validate the RS256 signature JWT token.

issuerURL, _ := url.Parse(os.Getenv("AUTH0_ISSUER_URL"))
audience := os.Getenv("AUTH0_AUDIENCE")

provider := jwks.NewCachingProvider(issuerURL, time.Duration(5*time.Minute))

jwtValidator, _ := validator.New(provider.KeyFunc,
    validator.RS256,
    issuerURL.String(),
    []string{audience},
)

jwtMiddleware := jwtmiddleware.New(jwtValidator.ValidateToken)
r.Use(adapter.Wrap(jwtMiddleware.CheckJWT))

The complete code looks like below.

package main

import (
    "log"
    "net/http"
    "net/url"
    "os"
    "time"

    adapter "github.com/gwatts/gin-adapter"

    jwtmiddleware "github.com/auth0/go-jwt-middleware/v2"
    "github.com/auth0/go-jwt-middleware/v2/jwks"
    "github.com/auth0/go-jwt-middleware/v2/validator"

    "github.com/gin-gonic/gin"
    "github.com/joho/godotenv"
)

type Product struct {
    ID    int     `json:"id"`
    Title string  `json:"title"`
    Code  string  `json:"code"`
    Price float32 `json:"price"`
}

func main() {
    err := godotenv.Load()
    if err != nil {
        log.Fatal("Error loading .env file")
    }

    r := gin.Default()

    issuerURL, _ := url.Parse(os.Getenv("AUTH0_ISSUER_URL"))
    audience := os.Getenv("AUTH0_AUDIENCE")

    provider := jwks.NewCachingProvider(issuerURL, time.Duration(5*time.Minute))

    jwtValidator, _ := validator.New(provider.KeyFunc,
        validator.RS256,
        issuerURL.String(),
        []string{audience},
    )

    jwtMiddleware := jwtmiddleware.New(jwtValidator.ValidateToken)
    r.Use(adapter.Wrap(jwtMiddleware.CheckJWT))

    r.GET("/products", func(c *gin.Context) {
        products := []Product{
            {ID: 1, Title: "Product 1", Code: "p1", Price: 100.0},
            {ID: 2, Title: "Product 2", Code: "p2", Price: 200.0},
            {ID: 3, Title: "Product 3", Code: "p3", Price: 300.0},
        }
        c.JSON(http.StatusOK, products)
    })

    // Listen and Server in 0.0.0.0:5000
    r.Run(":5000")
}

Verify Auth0 JWT

Create a test JWT from the API Test page. Call the API with the test token.

➜  auth0-go-gin-middleware  curl --location --request GET 'http://localhost:5000/products' \
--header 'authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6InVUT0ktNGhrSDBWNU9YUGxKV0xpXyJ9.eyJpc3MiOiJodHRwczovL3NpdmEtZGVtby1hcHAudXMuYXV0aDAuY29tLyIsInN1YiI6Inl2N1NDekVueUxTWGdMZ1d3b2pJODZvNk5ZMzh0cmNtQGNsaWVudHMiLCJhdWQiOiJodHRwczovL3Byb2R1Y3RzLWFwaS8iLCJpYXQiOjE2NDM0MzM4NTYsImV4cCI6MTY0MzUyMDI1NiwiYXpwIjoieXY3U0N6RW55TFNYZ0xnV3dvakk4Nm82TlkzOHRyY20iLCJndHkiOiJjbGllbnQtY3JlZGVudGlhbHMifQ.yH13H5XxLEABu3o8s2HZUs8Q9PXHeLGUcELQdlrYoClKP3k3B_WdUpaH2_c-UpA1ZjB_Is71-hSt3iqQN_OaMV_fFqpnt0qJQNXsoWSHBE5CzDDAclRlFf5XaWVbcA072rzUAtrJuPzHYf8kdR91243lJFA_13V5vlQuWxqFxas4FonVR5OLGcXYHqLfBI76DPfFaOBwOzefSYSI_jxKrrQtnux4Ktkqrgo7tpFckY6UfD6fXPeRvk4xLSb_SteAwcpCQrWJVyt7gTWv4_KkSNEyZduEbMTrnkU_jdQjHuKOLaTBc4t7t3MK_2_tmrda5xn0QXf-1y6P2M4zQu2mWA' | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   160  100   160    0     0   8045      0 --:--:-- --:--:-- --:--:-- 40000
[
  {
    "id": 1,
    "title": "Product 1",
    "code": "p1",
    "price": 100
  },
  {
    "id": 2,
    "title": "Product 2",
    "code": "p2",
    "price": 200
  },
  {
    "id": 3,
    "title": "Product 3",
    "code": "p3",
    "price": 300
  }
]

Conclusion

In this blog post, we walked through how to use Auth0 Golang middleware v2.0.0 to verify the JWT on both HS256 and RS256 signatures. You can extend this demo project to verify the custom claims in JWT and more. We will see in more detail in upcoming blog posts.

AWS EKS Connector - Manage all your Kubernetes Clusters in one place

Sivamuthu Kumar — Tue, 04 Jan 2022 03:30:11 +0000

Amazon EKS Connector is now generally available. With EKS Connector, you can now extend the EKS console to view your Kubernetes clusters outside AWS. What !?! Yes, now we can use the EKS console to visualize the Kubernetes clusters, both EKS and non-EKS clusters running in.

On-premises Kubernetes Cluster
Self-managed Clusters running on AWS EC2 instances
Clusters from other cloud providers

In today's blog, we will see how to connect the kubernetes clusters from my home lab and Azure / GCP Cloud providers along with EKS using eksctl. Once connected, we can see the cluster details, configurations, and workloads in one place on the EKS console.

Kubernetes Clusters

Home Lab

I have the home lab kubernetes cluster running on my mini pc that's running Linux. It was previously running on Raspberry Pi, and later I switched the lab to mini pc. I often flash Raspberry Pi for my IoT experiments; there are no apparent reasons. You can set up the kubernetes cluster on Raspberry Pi if you wish. I'm using K3s - lightweight kubernetes distribution designed for running production workloads in resource-constrained, IoT edge devices.

Azure Kubernetes Service

The Amazon EKS connector can also connect the Kubernetes clusters running on other cloud providers. I've set up the AKS (Azure Kubernetes Service) for this blog in the Azure environment.

EKS

I've EKS cluster along with the other clusters too. Amazon Elastic Kubernetes Service (Amazon EKS) is a managed container service to run and scale Kubernetes applications in AWS infrastructure.

Registering a Cluster

eksctl simplifies registering non-EKS clusters by creating the required AWS resources and generating Kubernetes manifests for EKS Connector to apply to the external cluster.

To register or connect a non-EKS Kubernetes cluster, run

➜ eksctl register cluster --name siva-home-lab --provider rancher

The supported providers are : EKS_ANYWHERE, ANTHOS, GKE, AKS, OPENSHIFT, TANZU, RANCHER, EC2, OTHER

When registering, eksctl creates three yaml files.

eks-connector.yaml - Deploys EKS connector agent
eks-connector-clusterrole.yaml - Cluster Role of cluster
eks-connector-console-dashboard-full-access-group.yaml - Console Dashboard Full Access

➜ eksctl register cluster --name siva-home-lab --provider rancher
2022-01-03 21:30:07 [ℹ]  creating IAM role "eksctl-20220103213007309381"
2022-01-03 21:30:17 [ℹ]  registered cluster "siva-home-lab" successfully
2022-01-03 21:30:17 [ℹ]  wrote file eks-connector.yaml to /Users/ksivamuthu/personal
2022-01-03 21:30:17 [ℹ]  wrote file eks-connector-clusterrole.yaml to /Users/ksivamuthu/personal
2022-01-03 21:30:17 [ℹ]  wrote file eks-connector-console-dashboard-full-access-group.yaml to /Users/ksivamuthu/personal
2022-01-03 21:30:17 [!]  note: "eks-connector-clusterrole.yaml" and "eks-connector-console-dashboard-full-access-group.yaml" give full EKS Console access to IAM identity "arn:aws:iam::495775103319:user/siva", edit if required; read https://docs.aws.amazon.com/eks/latest/userguide/connector-grant-access.html for more info
2022-01-03 21:30:17 [ℹ]  run `kubectl apply -f eks-connector.yaml,eks-connector-clusterrole.yaml,eks-connector-console-dashboard-full-access-group.yaml` before 07 Jan 22 02:30 UTC to connect the cluster

Apply the generated YAMLs in your cluster.

➜ kubectl apply -f eks-connector.yaml, \
            eks-connector-clusterrole.yaml, \
            eks-connector-console-dashboard-full-access-group.yaml

Dashboard - Workloads, Configuration & Tags

Once the cluster is registered and yamls are applied, you can view your non-EKS cluster in EKS Console.

You can see the workloads, configuration and manage tags of the non-eks clusters.

Let’s register the AKS cluster by repeating the registration steps.

Deregistering a cluster

To deregister cluster, you can use eksctl command.

➜ eksctl deregister cluster --name siva-home-lab --region us-east-1

It deregisters the cluster and we can delete the API server objects using kubectl

➜ eksctl deregister cluster --name siva-home-lab --region us-east-1
2022-01-03 21:56:39 [ℹ]  deleting IAM role "eksctl-20220103213007309381"
2022-01-03 21:56:39 [ℹ]  unregistered cluster "siva-home-lab" successfully
2022-01-03 21:56:39 [ℹ]  run `kubectl delete -f eks-connector.yaml,eks-connector-clusterrole.yaml,eks-connector-console-dashboard-full-access-group.yaml` on your cluster to remove EKS Connector resources

Conclusion

EKS Anywhere and EKS Connector are part of a clear play for businesses embracing hybrid cloud and private infrastructure setups. EKS Connector supports self-managed clusters on EC2, EKS Anywhere clusters running on-premises, and other Kubernetes clusters running outside of AWS to the EKS console. It’s great tool providing single pane of dashboard managing all of your Kubernetes cluster including EKS & Non EKS.

I'm Siva - working as Sr. Software Architect at Computer Enterprises Inc from Orlando. I'm an AWS Community builder, Auth0 Ambassador and I am going to write a lot about Cloud, Containers, IoT, and Devops. If you are interested in any of that, make sure to follow me if you haven’t already. Please follow me @ksivamuthu Twitter or check out my blogs at https://sivamuthukumar.com

Deploy .NET 6 API to AWS App Runner using AWS Copilot CLI

Sivamuthu Kumar — Sat, 30 Oct 2021 23:19:07 +0000

In this tutorial blog post, we are going to see how to deploy .NET 6 API to AWS App Runner using AWS Copilot CLI.

What is App Runner?

AWS App Runner is a fully managed service that makes it easy for developers to quickly deploy containerized web applications and APIs at scale and with no prior infrastructure experience required. Start with your source code or a container image. App Runner automatically builds and deploys the web application, and load balances traffic with encryption. App Runner also scales up or down automatically to meet your traffic needs. With App Runner, you have more time to focus on your applications rather than thinking about servers or scaling.

App Runner Features

The features of App Runner are

Autoscaling: starting and stopping as demand changes, between configurable min and max limits.
Load balancing: the service includes a transparent, non-configurable load balancer. The URL can be pointed to the custom domain.
SSL & Certificates: the deployed services will have HTTPS endpoints for applications with AWS-managed certificates. The certificates will be renewed when it's about to expire.
Build service: you can push your own images or let AWS build them for you from code.

AWS Copilot CLI

The AWS Copilot CLI is a tool for developers to build, release and operate production-ready containerized applications on AWS App Runner, Amazon ECS, and AWS Fargate. From getting started, pushing to staging, and releasing to production, Copilot can help manage the entire lifecycle of your application development.

aws / copilot-cli

The AWS Copilot CLI is a tool for developers to build, release and operate production ready containerized applications on AWS App Runner or Amazon ECS on AWS Fargate.

Setup a .NET 6 Minimal API

Let's get started. This section will create minimal API services using .NET and dockerize the application to deploy in AWS AppRunner using Copilot CLI.

Create a new web API project using dotnet
```
dotnet new web -n CoffeeService
```

The API code is like below that's by default generated by the template.

var builder = WebApplication.CreateBuilder(args);
var app = builder.Build();

app.MapGet("/", () => "Hello World!");

app.Run();

That's cool on seeing this minimal API (like how it's easy as developing Node.js code). Let's run locally and see whether we are getting this "Hello World" response.

Run the code
```
cd CoffeeService && dotnet run 
```

Execute the URL and check the response

➜  ~  curl http://localhost:5023/
Hello World!

Containerize the API

Create a file named Dockerfile in the directory containing the .csproj and open it in a text editor. Copy the contents below in the Dockerfile. In this Dockerfile, we are building the project and running it in the aspnet:6.0 runtime image.

FROM mcr.microsoft.com/dotnet/sdk:6.0 AS build
WORKDIR /src
COPY ["CoffeeService.csproj", "./"]
RUN dotnet restore "CoffeeService.csproj"
COPY . .
WORKDIR "/src/."
RUN dotnet build "CoffeeService.csproj" -c Release -o /app/build

FROM build AS publish
RUN dotnet publish "CoffeeService.csproj" -c Release -o /app/publish

FROM mcr.microsoft.com/dotnet/aspnet:6.0 AS base
WORKDIR /app
EXPOSE 5023
ENV ASPNETCORE_URLS=http://+:5023

WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "CoffeeService.dll"]

Build the docker image

docker build . -t coffee-service:latest

Run the docker image

docker run -p 5023:5023 coffee-service:latest

Execute the URL and check the response

➜  ~  curl http://localhost:5023/
Hello World!

Great. The .NET 6 minimal API is created and run from local and in the docker container. Now let's deploy into the AWS AppRunner using AWS Copilot CLI

Deploy into AWS AppRunner using AWS Copilot CLI

Run copilot init to set up the AWS AppRunner application for this service. The initialization tool is going to ask you the questions.

Use existing application or create a new application
Application name
Workload type

You can choose one of the workload types. In this demo, we are setting up the AWS App Runner.

- **Request-Driven Web Service  - (App Runner)**
- Load Balanced Web Service  - (Internet to ECS on Fargate)
- Backend Service - (ECS on Fargate)
- Worker Service - (Events to SQS to ECS on Fargate)
- Scheduled Job - (Scheduled event to State Machine to Fargate)

Service name and path of the Dockerfile to build
Once the ECR is set up, you can set up the environment to deploy.

Welcome to the Copilot CLI! We're going to walk you through some questions
to help you get set up with a containerized application on AWS. An application is a collection of
containerized services that operate together.

Use existing application: No
Application name: coffee-shop
Workload type: Request-Driven Web Service
Service name: coffee-service
Dockerfile: CoffeeService/Dockerfile
Ok great, we'll set up a Request-Driven Web Service named coffee-service in application coffee-shop listening on port 5023.

✔ Created the infrastructure to manage services and jobs under application coffee-shop..

✔ The directory copilot will hold service manifests for application coffee-shop.

✔ Wrote the manifest for service coffee-service at copilot/coffee-service/manifest.yml
Your manifest contains configurations like your container size and port (:5023).

✔ Created ECR repositories for service coffee-service..

All right, you're all set for local development.
Deploy: Yes

✔ Linked account 495775103319 and region us-east-1 to application coffee-shop..

✔ Proposing infrastructure changes for the coffee-shop-test environment.
- Creating the infrastructure for the coffee-shop-test environment.      [create complete]  [79.6s]
  - An IAM Role for AWS CloudFormation to manage resources               [create complete]  [16.2s]
  - An ECS cluster to group your services                                [create complete]  [9.0s]
  - Enable long ARN formats for the authenticated AWS principal          [create complete]  [4.6s]
  - An IAM Role to describe resources in your environment                [create complete]  [13.9s]
  - A security group to allow your containers to talk to each other      [create complete]  [5.9s]
  - An Internet Gateway to connect to the public internet                [create complete]  [16.1s]
  - Private subnet 1 for resources with no internet access               [create complete]  [19.3s]
  - Private subnet 2 for resources with no internet access               [create complete]  [19.6s]
  - Public subnet 1 for resources that can access the internet           [create complete]  [19.6s]
  - Public subnet 2 for resources that can access the internet           [create complete]  [19.6s]
  - A Virtual Private Cloud to control networking of your AWS resources  [create complete]  [16.1s]
✔ Created environment test in region us-east-1 under application coffee-shop.
Environment test is already on the latest version v1.6.1, skip upgrade.
[+] Building 0.7s (18/18) FINISHED
 => [internal] load build definition from Dockerfile                                                                                      0.0s
 => => transferring dockerfile: 588B                                                                                                      0.0s
 => [internal] load .dockerignore                                                                                                         0.0s
 => => transferring context: 374B                                                                                                         0.0s
 => [internal] load metadata for mcr.microsoft.com/dotnet/aspnet:6.0                                                                      0.5s
 => [internal] load metadata for mcr.microsoft.com/dotnet/sdk:6.0                                                                         0.5s
 => [internal] load build context                                                                                                         0.0s
 => => transferring context: 1.63kB                                                                                                       0.0s
 => [build 1/7] FROM mcr.microsoft.com/dotnet/sdk:6.0@sha256:96ce062b7e664999048b86198385fea1ddaff31d8d2ab5f7c42c0077678afeac             0.0s
 => [base 1/4] FROM mcr.microsoft.com/dotnet/aspnet:6.0@sha256:ed9b7dc3e8278a56be619b278762689565e1e21f61da51551fe028dc1d3a536f           0.0s
 => CACHED [base 2/4] WORKDIR /app                                                                                                        0.0s
 => CACHED [base 3/4] WORKDIR /app                                                                                                        0.0s
 => CACHED [build 2/7] WORKDIR /src                                                                                                       0.0s
 => CACHED [build 3/7] COPY [CoffeeService.csproj, ./]                                                                                    0.0s
 => CACHED [build 4/7] RUN dotnet restore "CoffeeService.csproj"                                                                          0.0s
 => CACHED [build 5/7] COPY . .                                                                                                           0.0s
 => CACHED [build 6/7] WORKDIR /src/.                                                                                                     0.0s
 => CACHED [build 7/7] RUN dotnet build "CoffeeService.csproj" -c Release -o /app/build                                                   0.0s
 => CACHED [publish 1/1] RUN dotnet publish "CoffeeService.csproj" -c Release -o /app/publish                                             0.0s
 => CACHED [base 4/4] COPY --from=publish /app/publish .                                                                                  0.0s
 => exporting to image                                                                                                                    0.0s
 => => exporting layers                                                                                                                   0.0s
 => => writing image sha256:74811373b140080ac1cf2f55c1e15a8dcdd9059f9a82d98a97a4f94a4e40e559                                              0.0s
 => => naming to 495775103319.dkr.ecr.us-east-1.amazonaws.com/coffee-shop/coffee-service                                                  0.0s

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
Login Succeeded
Using default tag: latest
The push refers to repository [495775103319.dkr.ecr.us-east-1.amazonaws.com/coffee-shop/coffee-service]
8cb51c566055: Pushed
...
e8b689711f21: Pushed
latest: digest: sha256:daeea40ed5b765e0ca27232964c529f3d6bb18eb8ccced8828a39c3153991c6a size: 1993
✔ Proposing infrastructure changes for stack coffee-shop-test-coffee-service
- Creating the infrastructure for stack coffee-shop-test-coffee-service           [create complete]    [246.5s]
  - An IAM Role for App Runner to use on your behalf to pull your image from ECR  [create complete]    [10.2s]
  - An IAM role to control permissions for the containers in your service         [create in progress]  [238.4s]
  - An App Runner service to run and manage your containers                       [create complete]    [225.4s]
✔ Deployed service coffee-service.
Recommended follow-up action:
    You can access your service at https://ptydisq8gp.us-east-1.awsapprunner.com over the internet.

Access the service at the URL.

➜ curl https://ptydisq8gp.us-east-1.awsapprunner.com
Hello World!

Copilot is doing the heavy lifting for us - the developers. So we can focus on the application. Copilot will build, push, and launch your container on AWS to ECS, Fargate, and AppRunner.

Setting up Automated Pipeline

The key principle of the devops process is "Ship small, Ship often." The process of deploying small features on a regular cadence is crucial to DevOps. As the team becomes more agile, we need to automate application releases as multiple developers; multiple services teams push the code into the source code repository.

AWS Copilot tool can help you in setting up the pipeline to automate application releases. You can run these commands to create an automated pipeline that builds and deploys the application on git push.

aws-apprunner-dotnet6-demo git:(main)  copilot pipeline init
1st stage: test
Repository URL: git@github.com:ksivamuthu/aws-apprunner-coffee-shop
✔ Wrote the pipeline manifest for aws-apprunner-coffee-shop at 'copilot/pipeline.yml'
The manifest contains configurations for your CodePipeline resources, such as your pipeline stages and build steps.
Update the file to add additional stages, change the branch to be tracked, or add test commands or manual approval actions.
✔ Wrote the buildspec for the pipeline's build stage at 'copilot/buildspec.yml'
The buildspec contains the commands to build and push your container images to your ECR repositories.
Update the build phase to unit test your services before pushing the images.

Required follow-up actions:
- Commit and push the buildspec.yml, pipeline.yml, and .workspace files of your copilot directory to your repository.
- Run `copilot pipeline update` to create your pipeline.

The copilot buildspec, pipeline yaml files are committed and pushed. The code build pipelines are set up to deploy automatically in the test environment.

Now, it's ready to bring more developers to start rocking. They don't need to install a copilot to deploy the machine's service as they are developing. The deployment steps are automated.

Let's push more code to see changes that get automatically deployed.

var builder = WebApplication.CreateBuilder(args);
var app = builder.Build();

app.MapGet("/", () => "Hello AWS AppRunner !!");
app.MapGet("/api/coffee", () => new List<dynamic> {
    new { CoffeeId = "cappucino", CoffeeName = "Cappucino" },
    new { CoffeeId = "latte", CoffeeName = "Latte" },
    new { CoffeeId = "mocha", CoffeeName = "Mocha" },
    new { CoffeeId = "americano", CoffeeName = "Americano" },
    new { CoffeeId = "macchiato", CoffeeName = "Macchiato" },
    new { CoffeeId = "frappe", CoffeeName = "Frappe" },
    new { CoffeeId = "corretto", CoffeeName = "Corretto" },
    new { CoffeeId = "affogato", CoffeeName = "Affogato" },
    new { CoffeeId = "filtercoffee", CoffeeName = "Filter Coffee" },
});

app.Run();

➜ aws-apprunner-dotnet6-demo git:(main) curl https://ptydisq8gp.us-east-1.awsapprunner.com
Hello AWS AppRunner !!

➜ aws-apprunner-dotnet6-demo git:(main) curl -s https://ptydisq8gp.us-east-1.awsapprunner.com/api/coffee | jq '.[0]'
{
  "coffeeId": "cappucino",
  "coffeeName": "Cappucino"
}

Test the Autoscaling

AWS App Runner automatically scales compute resources (instances) up or down for your App Runner application. Automatic scaling provides adequate request handling when incoming traffic is high and reduces your cost when traffic slows down. You can configure a few parameters to adjust autoscaling behavior for your service.

Settings – Here's what you can configure:
- Max concurrency – The maximum number of concurrent requests that an instance processes. When the number of concurrent requests exceeds this quota, App Runner scales up the service.
- Max size – The maximum number of instances that your service scales up to. At most this number of instances are actively serving traffic for your service.
- Min size – The minimum number of instances that App Runner provisions for your service. The service always has at least this number of provisioned instances. Some of them actively serve traffic. The rest of them (provisioned and inactive instances) stand by as a cost-effective compute capacity reserve, which is ready to be quickly activated. You pay for the memory usage of all provisioned instances. You pay for the CPU usage of only the active subset.
  
  App Runner temporarily doubles the number of provisioned instances during deployments to maintain the same old and new code capacity.

Let's create load testing with 100 concurrent requests and check the number of instances.

hey -z 1m -c 100 https://ptydisq8gp.us-east-1.awsapprunner.com/api/coffee

You can see the "active instances" count get increased based on the concurrent requests for autoscaling.

The source code repo of this demo

ksivamuthu / aws-apprunner-coffee-shop

What is App Runner?

App Runner Features

The features of App Runner are

Autoscaling: starting and stopping as demand changes, between configurable min and max limits.
Load balancing: the service includes a transparent, non-configurable load balancer. The URL can be pointed to the custom domain.
SSL & Certificates: the deployed services will have HTTPS endpoints for applications with AWS-managed certificates. The…

View on GitHub

Conclusion

AWS App Runner is a fully managed service that makes it easy for developers to quickly deploy containerized web applications and APIs at scale. It provides seamless "code-to-deploy" workflow for Node.js and Python runtime today and other runtimes using Dockerfile.Copilot also helps you to have your own customizable pipelines with just a few commands for your apps running AWS AppRunner.

I presented a talk on AWS App Runners in IndyAWS meetup. This session covers

AWS App Runner's features,
The key challenges AWS App Runner solves,
How to configure and integrate AWS App Runner with your source control to deploy your code in seconds
Live demo using AWS Copilot CLI, a tool that supports AWS App Runner.
Want a deeper dive? Download the presentation slides and youtube video of the talk.

Enforcing Policy as Code using Kyverno in Kubernetes

Sivamuthu Kumar — Wed, 06 Oct 2021 00:47:39 +0000

Hello all, In today's blog, we are going to learn about policy as code in Kubernetes using Kyverno. Let's get started.

What is Policy as Code?

Policy as Code (PaC) is the idea of writing code in a high-level language to manage and automate policies. The DevOps team can adopt the best practices by representing policies as code, such as version control, automated testing, and automated deployment. You can use these policies in audit or enforcement mode to monitor existing workloads, services for misconfiguration or prevent the misconfigurations applying in the cluster.

Policy as Code in Kubernetes

Policies could be established for multiple areas of your operational environments. You want your Kubernetes clusters to be reliable and secure and you want to control who has access to what. You also want to enforce rules for your kubernetes resources. There are specific things you want to enforce Kubernetes workloads from security, configuration, deployment best practices, operational concerns, governance and compliance requirements.

The three categories in the policy for kubernetes:

Standard policies - Best practices across the cluster in the organizations
- E.g: Requiring resources to specify the resource limits. Prevent workloads from running as root, etc
Organization policies - Enforce the policies specific to your organization
- E.g - Enforce the private image repository list to pull, Labels such as application name, environment to specify in workloads, policies with organization compliance and audit requirements.
Environment policies - Enforce the policies specific to environment
- E.g - Stricter security enforcement in production cluster

Kyverno - Policy Engine

Kyverno (Greek for "govern") is a policy engine designed specifically for Kubernetes. The features are

policies as Kubernetes resources in YAML (no new language to learn!)
validate, mutate, or generate any resource using Kustomize overlays
match resources using label selectors and wildcards
block non-conformant resources using admission controls, or report policy violations
test policies and validate resources using the Kyverno CLI, in your CI/CD pipeline, before applying them to your cluster

kyverno / kyverno

Cloud Native Policy Management

How does it work?

Kyverno runs as a dynamic admission controller in a Kubernetes cluster. Kyverno receives validating and mutating admission webhook HTTP callbacks from the kube-apiserver and applies matching policies to return results that enforce admission policies or reject requests.

Getting Started

The prerequisite for this tutorial is a functional kubernetes cluster. You can create the EKS cluster using the eksctl tool.

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: eks-k8s-policy-demo
  region: us-east-1

availabilityZones: 
  - us-east-1a
  - us-east-1b

managedNodeGroups:
  - name: eks-k8s-policy-demo-ng
    instanceType: t3.medium
    minSize: 1
    maxSize: 5

Policies and Usecases

The Kyverno team created the best practices and most used policies in this website here. There are three different policy types.

Validate
Mutate
Generate

We will see the demo on the below use cases and policies we can enforce for kubernetes deployments.

Enforce "application name" label in the pod
Require Limits and Requests
Add network policy

Enforce "Application Name" label in the pod

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-labels
spec:
  validationFailureAction: enforce
  rules:
  - name: check-for-labels
    match:
      resources:
        kinds:
        - Pod
    validate:
      message: "The label `app.kubernetes.io/name` is required."
      pattern:
        metadata:
          labels:
            app.kubernetes.io/name: "?*"

Let's see how this policy. works by creating the deployment in the pod. Apply the above policy in your cluster. Create a inflate deployment in the cluster without any labels.

➜  kyverno-demo git:(main) ✗  kubectl create deployment inflate --image=public.ecr.aws/eks-distro/kubernetes/pause:3.2
error: failed to create deployment: admission webhook "validate.kyverno.svc" denied the request:

resource Deployment/default/inflate was blocked due to the following policies

require-labels:
  autogen-check-for-labels: 'validation error: The label `app.kubernetes.io/name`
    is required. Rule autogen-check-for-labels failed at path /spec/template/metadata/labels/app.kubernetes.io/name/'

Now, let's create the deployment with app.kubernetes.io/name label.

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/name: inflate
  name: inflate
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: inflate
  template:
    metadata:
      labels:
        app.kubernetes.io/name: inflate
    spec:
      containers:
      - image: public.ecr.aws/eks-distro/kubernetes/pause:3.2
        name: pause

➜  kyverno-demo git:(main) ✗  kubectl apply -f deployment.yaml
deployment.apps/inflate created

Require Limits and Requests

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-requests-limits
spec:
  validationFailureAction: enforce
  rules:
  - name: validate-resources
    match:
      resources:
        kinds:
        - Pod
    validate:
      message: "CPU and memory resource requests and limits are required."
      pattern:
        spec:
          containers:
          - resources:
              requests:
                memory: "?*"
                cpu: "?*"
              limits:
                memory: "?*"

These policies are validation type policies. It validates the specific pattern in your kubernetes api objects.

Add network policy

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: add-networkpolicy
spec:
  validationFailureAction: enforce
  rules:
  - name: default-deny
    match:
      resources: 
        kinds:
        - Namespace
    generate:
      kind: NetworkPolicy
      name: default-deny
      namespace: "{{request.object.metadata.name}}"
      synchronize: true
      data:
        spec:
          # select all pods in the namespace
          podSelector: {}
          # deny all traffic
          policyTypes: 
          - Ingress
          - Egress

Monitoring - Dashboard

Kyverno has its metrics exposed through Prometheus metrics endpoint. You can scrape the kyverno metrics or display in grafana dashboard. Kyverno has the policy reporter UI that can target different channels. Checkout here.

kyverno / policy-reporter

Monitoring and Observability Tool for the PolicyReport CRD with an optional UI.

Shift Left

You can validate the policies before applying the cluster using cli. I've integrated kyverno cli action in github before deploying yaml files in kubernetes cluster.

ksivamuthu / kyverno-policy-demo

- name: Validate policy
        uses: gbaeke/kyverno-cli@v1
        with:
          command: |
            kyverno apply ./policies --resource=./k8s/2048.yaml

After fixing the YAML files

Conclusion

You can improve the security posture for your clusters with policies as code using Kyverno. No need to learn new language to create or manager policies. It works with your existing tools such as git, kustomize, kubectl etc.

I'm Siva - working as Sr. Software Architect at Computer Enterprises Inc from Orlando. I'm an AWS Community builder, Auth0 Ambassador and I am going to write a lot about Cloud, Containers, IoT, and Devops. If you are interested in any of that, make sure to follow me if you haven’t already. Please follow me @ksivamuthu Twitter or check out my blogs at https://blog.sivamuthukumar.com!

Karpenter - Scaling Nodes Seamlessly in AWS EKS

Sivamuthu Kumar — Fri, 24 Sep 2021 02:49:51 +0000

Hello everyone !! If you are running your workloads on the AWS EKS cluster, you may explore the rules and limitations of node group scaling to provision the deployments dynamically. This blog will explore the node lifecycle management solutions AWS Lab's Karpenter, an alternative approach to the frequently used Cluster Autoscaler solution.

Kubernetes Autoscaling

Autoscaling allows you to dynamically adjust to demand without manual intervention through metrics or events. Without autoscaling, there will be considerable efforts to provision the (scaling up or down) resources. When the running conditions change, and optimal resource utilization and managing the cloud spending is challenging. The cluster is always running at peak capacity to ensure availability or not meeting peak demand as they don't have enough resources.

When it comes to the Kubernetes Autoscaling, there are two different layers,

Pod Level Autoscaling (Horizontal - HPA, and Vertical - HPA)
Node Level Autoscaling

Pod Level Autoscaling

Horizontal Pod Autoscaling (Scaling out) - dynamically increase or decrease the number of running pods per your application's usage changes.

Vertical Pod Autoscaling (Scaling up) - scale the given deployments vertically within a cluster by reconciling the pods' size ( CPU or memory targets) based on their current usage and the desired target.

HPA and VPA essentially make sure that all of the services running in your cluster can dynamically handle the demand.

Node Level Autoscaling

Node Level autoscaling solves the issue - to scale the nodes in the cluster when the existing nodes are overloaded or pending to be scheduled with newly scaled pods or scale down when the nodes are underutilized.

There is already an industry-adopted, open-source, and vendor-neutral tool - Cluster Autoscaler that automatically adjusts the cluster size (by adding or removing nodes) based on the presence of pending pods and node utilization metrics. It uses the existing cloud building blocks (Autoscaling Group on AWS) for scaling. The challenges in the cluster autoscaler are the limitations on node groups, and the scaling is tightly bound to the scheduler.

kubernetes / autoscaler

Autoscaling components for Kubernetes

Karpenter

Karpenter is a node lifecycle management solution - incubating in AWS Labs, OSS, and vendor-neutral. It observes incoming pods and launches the right instances for the situation. Instance selection decisions are intent-based and driven by the specification of incoming pods, including resource requests and scheduling constraints.

aws / karpenter-provider-aws

Karpenter is a Kubernetes Node Autoscaler built for flexibility, performance, and simplicity.

Karpenter is an open-source node provisioning project built for Kubernetes Karpenter improves the efficiency and cost of running workloads on Kubernetes clusters by:

Watching for pods that the Kubernetes scheduler has marked as unschedulable
Evaluating scheduling constraints (resource requests, nodeselectors, affinities, tolerations, and topology spread constraints) requested by the pods
Provisioning nodes that meet the requirements of the pods
Removing the nodes when the nodes are no longer needed

Come discuss Karpenter in the #karpenter channel, in the Kubernetes slack or join the Karpenter working group bi-weekly calls. If you want to contribute to the Karpenter project, please refer to the Karpenter docs.

Check out the Docs to learn more.

Talks

03/19/2024 Harnessing Karpenter: Transforming Kubernetes Clusters with Argo Workflows
12/04/2023 AWS re:Invent 2023 - Harness the power of Karpenter to scale, optimize & upgrade Kubernetes
09/08/2022 Workload Consolidation with Karpenter
05/19/2022 Scaling K8s Nodes Without Breaking the Bank or…

View on GitHub

How does it work?

Observes the pod resource requests of unscheduled pods
Direct provision of Just-in-time capacity of the node. (Groupless Node Autoscaling)
Terminating nodes if outdated
Reallocating the pods in nodes for better resource utilization

Karpenter has two control loops that maximize the availability and efficiency of your cluster.

Allocator - Fast-acting controller ensuring that pods are scheduled as quickly as possible
Reallocator - Slow-acting controller replaces nodes as pods capacity shifts over time.

Getting started

In this section, we will quickly see the node lifecycle scenarios using Karpenter in an AWS EKS cluster. Create necessary IAM roles for Karpenter autoscaler with the cloud formation template and Create EKS cluster with the below config file using eksctl. Please refer to the documentation here.

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: eks-karpenter-demo
  region: us-east-1

availabilityZones: 
  - us-east-1a
  - us-east-1b

managedNodeGroups:
  - name: eks-karpenter-demo-ng
    instanceType: t3.medium
    minSize: 1
    maxSize: 5

You need to enable the service account and auth-config map accounts to the Karpenter. Please refer to the document here.

Install the karpenter helm chart.

helm repo add karpenter https://awslabs.github.io/karpenter/charts
helm repo update
helm upgrade --install karpenter karpenter/karpenter --namespace karpenter \
  --create-namespace --set serviceAccount.create=false --version 0.3.3

Configure the Karpenter Provisioner

Configure the Karpenter provisioner as below. Please check the provider spec for more details.

apiVersion: karpenter.sh/v1alpha3
kind: Provisioner
metadata:
  name: default
spec:
  cluster:
    name: eks-karpenter-demo
    endpoint: <CLUSTER_ENDPOINT>
  instanceTypes:
    - t3.medium    
  ttlSecondsAfterEmpty: 30

Deployment

Let's do the deployment to check the launching capacity and terminating capacity features.

kubectl create deployment inflate --image=public.ecr.aws/eks-distro/kubernetes/pause:3.2

Provisioning Nodes

Scale the deployment and check out the logs in the Karpenter controller.

kubectl scale deployment inflate --replicas=10

Check the logs of the karpenter controller

➜  eks-karpenter-demo git:(main) kubectl logs -f -n karpenter $(kubectl get pods -n karpenter -l karpenter=controller -o name)
2021-09-23T04:46:11.280Z        INFO    controller.allocation.provisioner/default       Starting provisioning loop      {"commit": "bc99951"}
2021-09-23T04:46:11.280Z        INFO    controller.allocation.provisioner/default       Waiting to batch additional pods        {"commit": "bc99951"}
2021-09-23T04:46:12.452Z        INFO    controller.allocation.provisioner/default       Found 9 provisionable pods      {"commit": "bc99951"}
2021-09-23T04:46:13.689Z        INFO    controller.allocation.provisioner/default       Computed packing for 9 pod(s) with instance type option(s) [t3.medium]  {"commit": "bc99951"}
2021-09-23T04:46:16.228Z        INFO    controller.allocation.provisioner/default       Launched instance: i-0174aa47fe6d1f7b4, type: t3.medium, zone: us-east-1b, hostname: ip-192-168-116-109.ec2.internal    {"commit": "bc99951"}
2021-09-23T04:46:16.265Z        INFO    controller.allocation.provisioner/default       Bound 9 pod(s) to node ip-192-168-116-109.ec2.internal  {"commit": "bc99951"}
2021-09-23T04:46:16.265Z        INFO    controller.allocation.provisioner/default       Watching for pod events {"commit": "bc99951"}

The allocation controller listens for pods changes. It launched a new instance and bound the provision-able pods into the new nodes by working with kube-scheduler.

The provisioning time is fast compared to other node management solutions. The other node management solutions usually take 3 min to 6 min for the node to be available. After deploying the pods, the instances are immediately created and binding. The provisioner decides to launch a new instance within a second, and the node joins the cluster for under 60 seconds. Within 60 seconds, the nodes are available to cluster for running pods.

You can configure the instance types, capacity type, os, architecture, and other provisioner spec fields.

Terminating Nodes

Now, delete the deployment inflate. After 30 seconds (ttlSecondsAfterEmpty - Termination grace period), Karpenter should terminate the empty nodes - cordon & drain by listening to the rebalance and termination events.

2021-09-23T04:46:18.953Z        INFO    controller.allocation.provisioner/default       Watching for pod events {"commit": "bc99951"}
2021-09-23T04:49:05.805Z        INFO    controller.Node Added TTL to empty node ip-192-168-116-109.ec2.internal {"commit": "bc99951"}
2021-09-23T04:49:35.823Z        INFO    controller.Node Triggering termination after 30s for empty node ip-192-168-116-109.ec2.internal {"commit": "bc99951"}
2021-09-23T04:49:35.849Z        INFO    controller.Termination  Cordoned node ip-192-168-116-109.ec2.internal   {"commit": "bc99951"}
2021-09-23T04:49:36.521Z        INFO    controller.Termination  Deleted node ip-192-168-116-109.ec2.internal    {"commit": "bc99951"}

Next Steps

Autoscaling nodes are always challenging. Karpenter addresses key areas of challenges by eliminating Node Group and directly provision nodes. Karpenter is easy to configure, high-performance portable solution, and vendor-agnostic. It scales seamlessly working alongside native kube-scheduler and efficiently responds to dynamic resource requests.

Check out the AWS Labs Karpenter Roadmap. It's still in beta. In the year 2021, Karpenter is going to focus on covering the majority of known use cases and plan to rigorously test it for scale and performance.

I'm Siva - working as Sr. Software Architect at Computer Enterprises Inc from Orlando. I'm an AWS Community builder, Auth0 Ambassador and I am going to write a lot about Cloud, Containers, IoT, and Devops. If you are interested in any of that, make sure to follow me if you haven’t already. Please follow me @ksivamuthu Twitter or check out my blogs at https://blog.sivamuthukumar.com