Forem: Devtron

How to Backup and Restore Kubernetes clusters using Velero

Devtron — Mon, 14 Oct 2024 11:02:00 +0000

Ensuring the security and recovery of applications and data is important in the Kubernetes world. A powerful tool that can help achieve this is Velero, a versatile backup and recovery solution designed specifically for Kubernetes clusters. In this guide, we'll cover the process of securing your Kubernetes Cluster with Velero, providing you with peace of mind and protection against unforeseen events.

Understanding Velero

Velero, formerly known as Heptio Ark, is an open-source tool that simplifies backup, recovery, and migration for Kubernetes cluster resources and regular containers. It allows users to create scheduled or temporary backups of their resources and applications, ensuring data integrity and recovery in the event of failure, damage, or improper deletion.

All Velero operations (on-demand backup, scheduled backup, restore) are private, defined using the Kubernetes Custom Resource Definition (CRD), and stored elsewhere. Velero also includes a controller that manages dedicated resources to perform backup, restore, and all related tasks.

Velero is ideal for implementing disaster recovery and taking snapshots of the application state before performing cluster operations such as upgrades.

Why Backup Your Kubernetes Cluster?

It is important to implement a backup strategy to reduce the risks associated with data loss or corruption. By using Velero to regularly backup your Kubernetes Application Stack, you can:

Improved Testability: Regularly scheduled backups with Velero allow you to create isolated environments for testing purposes. You can restore a specific point-in-time backup of your Application stack to test new features, configurations, or disaster recovery procedures without impacting your production environment.
Reduced Downtime: In the event of an accidental deletion, configuration drift, or even a security breach, restoring from a recent Velero backup can significantly reduce downtime compared to rebuilding your K8s Application stack from scratch. This translates to faster recovery times and minimized disruptions for your users.
Granular Backups and Restores: Velero offers flexibility when it comes to backups. You can choose to back up the entire Application stack or specific components like the Devtron CRDs (Custom Resource Definitions) or specific namespaces and their configuration data. This allows for granular restores, where you can recover only the affected part of the stack instead of the entire system.
Compliance and Auditability: For organizations with strict compliance requirements, Velero backups provide a verifiable audit trail. You can track backup versions, timestamps, and success/failure logs, demonstrating adherence to data retention policies and regulations.
Disaster Recovery Across Environments: Velero supports backups to various cloud providers and on-premises storage solutions. This enables you to restore your K8s Cluster to a completely different environment in case of a disaster that renders your primary cluster unusable. This provides an additional layer of protection and ensures business continuity.

Workflow of Cluster Backup using Velero

Velero gets installed on the cluster with the given configurations. While backing up Velero creates the .tar files of the backup and pushes them onto the storage provider. For restoring the backups, Velero searches for the .tar files on the given storage provider, pulls it into the target cluster, and applies it to the cluster as you can see in the [Fig 1]

Prerequisites for Cluster Backup Using Velero

Let’s take the example of a cluster where Devtron is running. We would like to take the backup of the cluster and restore the backup in case of any disaster. For this demo, we would be using AWS S3 to store the backup and then restoring the backup in the target cluster using the backup pushed at S3.

Before getting our hands dirty, we need to install the CLI first.

Installation

Linux:

Install the Velero CLI client from the official GitHub page of Velero i.e. https://github.com/vmware-tanzu/velero on the CLI from where you have access to the Kubernetes cluster.

After downloading, extract the tar file and add Velero to the bin.

sudo mv /Users/demo-user/velero-v1.14.0-linux-amd64/velero  /usr/local/bin

For macOS:

You can use homebrew - brew install velero

Configure Velero for Cluster Backup & Restore

Let's configure the Velero CLI that we would be using for taking the backup of the Kubernetes cluster and the same CLI would be used for restoring the backup in the target cluster.

Step-1: Create a directory named Velero. Follow the commands below to navigate to the directory named velero

mkdir velero
cd ./velero

Create a file named Velero-creds and add the following

[default]
aws_access_key_id = < s3_storage_access_key_id> 
aws_secret_access_key = < s3_storage_secret_access >

Step-2: Install the Velero client with the following configurations

velero install \
--provider aws \
--plugins velero/velero-plugin-for-aws:v1.10.0 \
--bucket k8s-backup \
--backup-location-config region=ap-southeast-1 \
--snapshot-location-config region=ap-southeast-1 \
--secret-file ./velero-creds

You can check the installation by running the following command.

kubectl get all -n velero

Now the Velero has been installed successfully on the cluster and has been configured.

Cluster Backup & Restore

After configuring the Velero CLI, now let's take the backup of the cluster, and restore the backup in the target cluster.

Backup

Now as the Velero is configured we need to run a command to take the backup of our Kubernetes cluster.

Run the following command:

velero backup create k8s-backup

This command will create a backup of the Kubernetes cluster and store it in the storage option provided with the given name i.e k8s-backup

You can also take backups of specific namespaces using the following command

velero backup create <backup-name> --include-namespaces <namespace>

Once the backup is completed you can see the tar files of your backup on your S3 bucket. Check the backup by running the following command

Velero backup describe <backup name>

Restore

Now the backup is ready and can be restored by going through the same process of installing Velero and running the following commands.

Note: If you restore the backup on the same cluster where the Velero is configured, there is no need to configure it again.

velero restore create –from-backup devtroncd

There is an option to include the PVCs with the backup using --csi-snapshot-timeout flag

velero backup create nginx-backup --include-namespacesnginx-example --csi-snapshot-timeout 20m

In this way, you can recover the K8s cluster using Velero. For more operations, you can go to the documentation or use the command: velero --help

If you have any questions, feel free to join our vibrant Discord Community and share your questions if you have any.

12 Tools that will make Kubernetes management easier in 2024

Devtron — Thu, 10 Oct 2024 12:01:00 +0000

Kubernetes, the revolutionary container orchestration platform, has empowered developers to build, deploy, and scale applications with unparalleled flexibility and efficiency for a decade. But let's be honest, Kubernetes' sheer scale and complexity can sometimes feel overwhelming, even for seasoned DevOps engineers. You're still battling tangled deployments, resource conflicts, and security concerns frequently. Developers may face distractions from core application development due to manual operational tasks and the steep learning curve.

But fear not, Intrepid CloudNative Warriors! This blog post is your guide to navigating the intricate world of Kubernetes with ease. We'll explore 12 essential tools that streamline workflows, boost efficiency, and make managing your Kubernetes management easier in 2024.

But Wait! Why Simplify Kubernetes - Management?

Let's face it, modern applications are demanding. You're juggling containerized workloads, scaling for peak usage, and ensuring everything runs smoothly. This is where efficient management tools come in, offering these crucial benefits:

Reduced Complexity: No more wrestling with Kubectl commands or endless configuration files. These tools offer intuitive interfaces and automated processes that make your life easier.
Increased Efficiency: Let's be real, you're a busy person. These tools automate repetitive tasks and streamline workflows, freeing you up to focus on problem-solving & innovation.
Enhanced Security: Security is non-negotiable. Dedicated security tools proactively scan for vulnerabilities, enforce policies, and help you keep your applications and infrastructure safe from harm.
Improved Cost Optimization: You want to make the most of your resources and cost analyzers and autoscale help you optimize resource utilization, preventing over-provisioning and saving you money.

Now that we've recognized the significance, let's delve into 12 essential tools that streamline Kubernetes management.

1. KEDA

Keda (Kubernetes Event-Driven Autoscaling) is an event-driven autoscale for Kubernetes workloads. Simply defined, it can scale an application based on the number of events needing to be handled.

Installing KEDA

KEDA has well-documented steps for installation in their documentation. You can install it with Helm, Operator Hub, or YAML declarations. In this blog, let's go with the helm.

Step 1: Add Helm Repo



helm repo add kedacore https://kedacore.github.io/charts

Step 2: Update Helm Repo



helm repo update

Step 3: Install KEDA in keda Namespace



helm install keda kedacore/keda --namespace keda --create-namespace

Use Cases

KEDA is a classic example of autoscaling based on different metrics. When you want to autoscale your application beyond the resource metrics like CPU/ Memory, you can use KEDA. It listens to specific events such as messages from message queues, HTTP requests, custom Prometheus metrics, Kafka lag, etc. To deep dive into KEDA, check out this blog.

2. Karpenter

Built in AWS, Karpenter is a high-performance, flexible, open-source Kubernetes cluster auto-scaler. One of its key features is the ability to launch EC2 instances based on specific workload requirements such as storage, compute, acceleration, and scheduling needs.

Installation

Install Karpenter in the Kubernetes cluster using Helm charts. But before doing this, you must ensure enough computing capacity is available. Karpenter requires permissions to provision compute resources that are based on the cloud provider you have chosen.

Step 1: Install Utilities

Karpenter can be installed in clusters using a Helm chart. Install these tools before proceeding:

AWS CLI
kubectl - the Kubernetes CLI
eksctl (>= v0.180.0) - the CLI for AWS EKS
helm - the package manager for Kubernetes

Configure the AWS CLI with a user that has sufficient privileges to create an EKS cluster. Verify that the CLI can authenticate properly by running aws sts get-caller-identity.

Step 2: Set Environment Variables

After installing the dependencies, set the Karpenter namespace, version and Kubernetes version as follows.



export KARPENTER_NAMESPACE="kube-system"
export KARPENTER_VERSION="0.37.0"
export K8S_VERSION="1.30"

Then set the following environment variables which would be further used for creating an EKS cluster.



export AWS_PARTITION="aws" # if you are not using standard partitions, you may need to configure to aws-cn / aws-us-gov
export CLUSTER_NAME="${USER}-karpenter-demo"
export AWS_DEFAULT_REGION="us-west-2"
export AWS_ACCOUNT_ID="$(aws sts get-caller-identity --query Account --output text)"
export TEMPOUT="$(mktemp)"
export ARM_AMI_ID="$(aws ssm get-parameter --name /aws/service/eks/optimized-ami/${K8S_VERSION}/amazon-linux-2-arm64/recommended/image_id --query Parameter.Value --output text)"
export AMD_AMI_ID="$(aws ssm get-parameter --name /aws/service/eks/optimized-ami/${K8S_VERSION}/amazon-linux-2/recommended/image_id --query Parameter.Value --output text)"
export GPU_AMI_ID="$(aws ssm get-parameter --name /aws/service/eks/optimized-ami/${K8S_VERSION}/amazon-linux-2-gpu/recommended/image_id --query Parameter.Value --output text)"

Step 3: Create a Cluster

The following configs will create an EKS cluster with the user configured in aws-cli having the relevant permissions to create an EKS cluster.



curl -fsSL https://raw.githubusercontent.com/aws/karpenter-provider-aws/v"${KARPENTER_VERSION}"/website/content/en/preview/getting-started/getting-started-with-karpenter/cloudformation.yaml  > "${TEMPOUT}" \
&& aws cloudformation deploy \
  --stack-name "Karpenter-${CLUSTER_NAME}" \
  --template-file "${TEMPOUT}" \
  --capabilities CAPABILITY_NAMED_IAM \
  --parameter-overrides "ClusterName=${CLUSTER_NAME}"

eksctl create cluster -f - <<EOF
---
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: ${CLUSTER_NAME}
  region: ${AWS_DEFAULT_REGION}
  version: "${K8S_VERSION}"
  tags:
    karpenter.sh/discovery: ${CLUSTER_NAME}

iam:
  withOIDC: true
  podIdentityAssociations:
  - namespace: "${KARPENTER_NAMESPACE}"
    serviceAccountName: karpenter
    roleName: ${CLUSTER_NAME}-karpenter
    permissionPolicyARNs:
    - arn:${AWS_PARTITION}:iam::${AWS_ACCOUNT_ID}:policy/KarpenterControllerPolicy-${CLUSTER_NAME}

iamIdentityMappings:
- arn: "arn:${AWS_PARTITION}:iam::${AWS_ACCOUNT_ID}:role/KarpenterNodeRole-${CLUSTER_NAME}"
  username: system:node:{{EC2PrivateDNSName}}
  groups:
  - system:bootstrappers
  - system:nodes
  ## If you intend to run Windows workloads, the kube-proxy group should be specified.
  # For more information, see https://github.com/aws/karpenter/issues/5099.
  # - eks:kube-proxy-windows

managedNodeGroups:
- instanceType: m5.large
  amiFamily: AmazonLinux2
  name: ${CLUSTER_NAME}-ng
  desiredCapacity: 2
  minSize: 1
  maxSize: 10

addons:
- name: eks-pod-identity-agent
EOF

export CLUSTER_ENDPOINT="$(aws eks describe-cluster --name "${CLUSTER_NAME}" --query "cluster.endpoint" --output text)"
export KARPENTER_IAM_ROLE_ARN="arn:${AWS_PARTITION}:iam::${AWS_ACCOUNT_ID}:role/${CLUSTER_NAME}-karpenter"

echo "${CLUSTER_ENDPOINT} ${KARPENTER_IAM_ROLE_ARN}"

Unless your AWS account has already been onboarded to EC2 Spot, you will need to create the service-linked role to avoid the ServiceLinkedRoleCreationNotPermitted error.



aws iam create-service-linked-role --aws-service-name spot.amazonaws.com || true
# If the role has already been successfully created, you will see:
# An error occurred (InvalidInput) when calling the CreateServiceLinkedRole operation: Service role name AWSServiceRoleForEC2Spot has been taken in this account, please try a different suffix.

Step 4: Install Karpenter



# Logout of helm registry to perform an unauthenticated pull against the public ECR
helm registry logout public.ecr.aws

helm upgrade --install karpenter oci://public.ecr.aws/karpenter/karpenter --version "${KARPENTER_VERSION}" --namespace "${KARPENTER_NAMESPACE}" --create-namespace \
  --set "settings.clusterName=${CLUSTER_NAME}" \
  --set "settings.interruptionQueue=${CLUSTER_NAME}" \
  --set controller.resources.requests.cpu=1 \
  --set controller.resources.requests.memory=1Gi \
  --set controller.resources.limits.cpu=1 \
  --set controller.resources.limits.memory=1Gi \
  --wait

Once the installation is done, you can create NodePool and define the instance family, architecture, and start using Karpenter as your cluster autoscaler. For detailed installation information and its usage, feel free to refer to its documentation.

Use Cases

Karpenter is used to automatically provision and optimize Kubernetes cluster resources, ensuring efficient and cost-effective scaling. It dynamically adjusts node capacity based on workload demands, reducing over-provisioning and underutilization, thus enhancing performance and lowering cloud infrastructure costs. Check out this blog to understand in-depth about Karpenter and Cluster Autocaler.

3. Devtron

Devtron is a tool integration platform for Kubernetes and enables swift app containerization, seamless Kubernetes deployment, and peak performance optimization. It deeply integrates with products across the lifecycle of microservices i.e., CI/CD, security, cost, debugging, and observability via an intuitive web interface.

Devtron helps you to deploy, observe, manage & debug the existing Helm apps in all your clusters.

Installation

Run the following command to install the latest version of Devtron along with the CI/CD module:



helm repo add devtron https://helm.devtron.ai



helm repo update devtron



helm install devtron devtron/devtron-operator \
--create-namespace --namespace devtroncd \
--set installer.modules={cicd}

Check out the complete guide here. If you have questions, please let us know on our discord channel.

Use Cases

Devtron simplifies Kubernetes adoption by addressing key challenges, making it easier to deploy, monitor, observe, and debug applications at scale.

Here's how Devtron helps:

1. Simplifying the Adoption Process:

Single Pane of Glass: Provides a unified view of all Kubernetes resources, enabling easy navigation and understanding of cluster components.
Real-Time Application Status Monitoring: Displays the health and status of applications in real-time, highlighting potential issues and unhealthy components.
Simplified Debugging: Offers tools like event logs, pod logs, and interactive shells for debugging issues within the Kubernetes environment.
Containerization Made Easy: Provides templates and options for building container images, simplifying the containerization process for various frameworks and languages.

2. Streamlining Tool Integration:

Helm Marketplace: Integrates with the Helm chart repository to easily deploy and manage various Kubernetes tools.
Built-in Integrations: Offers native integrations with popular tools like Grafana, Trivy, and Clair for enhanced functionality.

3. Simplifying Multi-Cluster/Cloud Workloads:

Centralized Visibility: Provides a unified view of applications across multiple clusters and cloud environments, enabling consistent management.
Environment-Specific Configurations: Allows setting environment-specific configurations, making it easier to manage applications in diverse environments.

4. Simplifying DevSecOps:

Fine-Grained Access Control: Enables granular control over user permissions for Kubernetes resources, ensuring secure access.
Security Scanning and Policies: Offers built-in security scanning with Trivy and allows configuring policies to enforce security best practices.

If you liked what Devtron is solving, do give it a Star ⭐️ on GitHub.

4. K9s

K9s is a terminal-based UI to interact with your Kubernetes clusters. This project aims to make it easier to navigate, observe, and manage your deployed applications in the wild. K9s continually watch Kubernetes for changes and offer subsequent commands to interact with your observed resources.

Installation

K9s is available on Linux, macOS, and Windows platforms. You can get the latest binaries for different architectures and operating systems from the releases on GitHub.

MacOS/ Linux



 # Via Homebrew
 brew install derailed/k9s/k9s

Windows



# Via chocolatey
choco install k9s

For other ways of installation, feel free to check out the documentation of K9s.

Use Cases

K9s make it much easier as compared to other Kubernetes clients like kubectl to manage and orchestrate applications on Kubernetes. You get a terminal-based GUI which helps you manage your resources,

Resource Monitoring and Visibility: It provides continuous monitoring of your Kubernetes cluster, offering a clear view of resource statuses. It helps you understand your K8s cluster by displaying information about pods, deployments, services, nodes, and more. With K9s, you can easily navigate through cluster resources, ensuring better visibility and awareness.
Resource Interaction and Management: K9s allows you to interact with resources directly from the terminal. You can view, edit, and delete resources without switching to a separate management tool. Common operations include scaling deployments, restarting pods, and inspecting logs. You can also initiate port-forwarding to access services running within pods.
Namespace Management: This lets you focus on specific namespaces within your cluster. You can switch between namespaces seamlessly, making it easier to work with isolated environments. By filtering resources based on namespaces, you can avoid clutter and stay organized.
Advanced Features: It also offers advanced capabilities, such as opening a shell in a container directly from the UI. It supports context switching between different clusters, making it convenient for multi-cluster environments. Additionally, K9s integrate with vulnerability scanning tools, enhancing security practices.

5. Winter Soldier

Winter Soldier is an open-source tool from Devtron, it enables time-based scaling for Kubernetes workloads. The time-based scaling with Winter Soldier helps us to reduce the cloud cost, it can be deployed to execute things such as:

Batch deletion of the unused resources
Scaling of Kubernetes workloads

Installation

If you want to dive deeper into it, please check the following resources -https://devtron.ai/blog/winter-soldier-scale-down-your-infrastructure-in-the-easiest-possible-way

Give it Star on the Github if you like the project: https://github.com/devtron-labs/winter-soldier

Use Cases

Winter Soldier is a valuable tool for anyone who wants to:

Optimize Cloud Costs: By automatically scaling Kubernetes workloads based on time, Winter Soldier helps reduce unnecessary resource usage, lowering cloud bills.
Automate Routine Tasks: Tasks like deleting unused resources or scaling workloads at specific times can be automated, freeing up time for other initiatives.
Improve Resource Utilization: By ensuring resources are only allocated when needed, Winter Soldier maximizes resource utilization and improves overall efficiency.
Implement Time-Based Scaling: It's ideal for scenarios where workloads have predictable usage patterns (e.g., a website that experiences heavy traffic during specific hours) or when resources need to be adjusted based on time.

Examples:

E-commerce Website: Scale up resources during peak shopping hours and scale down during off-peak periods to reduce costs.
Data Processing Jobs: Schedule resource scaling for batch processing jobs that run only during specific time windows.
Development and Testing Environments: Automatically scale down development and testing environments after hours to minimize resource usage.

Key Benefits:

Cost Reduction: Optimizing resource utilization translates to lower cloud bills.
Improved Efficiency: Automating resource management frees up time for other tasks.
Enhanced Reliability: By ensuring resources are allocated appropriately, Winter Soldier helps improve the reliability of Kubernetes applications.

6. Silver Surfer

Currently, there is no easy way to upgrade Kubernetes objects in case of Kubernetes upgrade. It's a tedious task to know whether the current ApiVersion of the Object is Removed, Deprecated, or Unchanged. It provides details of issues with the Kubernetes object in case they are migrated to a cluster with a newer Kubernetes version.

Installation

Just with a few commands, it's ready to serve your cluster.



git clone https://github.com/devtron-labs/silver-surfer.git
cd silver-surfer
go mod vendor
go mod download
make

It's done. A bin directory might have been created with the binary ready-to-use ./kubedd command.

It categorizes Kubernetes objects based on changes in ApiVersion. Categories are:

Removed ApiVersion
Deprecated ApiVersion
Newer ApiVersion
Unchanged ApiVersion

Within each category it identifies the migration path to the newer API Version, possible paths are:

It cannot be migrated as there are no common ApiVersions between the source and target Kubernetes version
It can be migrated but has some issues which need to be resolved
It can be migrated with just an ApiVersion change

This activity is performed for both current and new ApiVersion.

Check out the Github repo and give it a star ⭐️: https://github.com/devtron-labs/silver-surfer

Use Cases

Pre-Upgrade Planning: Silver Surfer helps you identify potential issues before the upgrade, giving you time to plan and resolve them.
Streamlined Upgrade Process: The tool provides detailed guidance, minimizing downtime and errors during the upgrade.
Kubernetes Object Management: Silver Surfer provides greater visibility into the compatibility of your objects with different Kubernetes versions, aiding in managing your cluster.

Key Benefits

Reduced Upgrade Complexity: Simplifies the Kubernetes upgrade process, reducing stress and the potential for errors.
Improved Uptime: Minimizes downtime during the upgrade process.
Enhanced Cluster Management: Provides a better understanding of your Kubernetes objects and their compatibility with different versions.

7. Trivy

Trivy is a simple and comprehensive vulnerability scanner for containers and other artifacts. A software vulnerability is a glitch, flaw, or weakness present in the software or in an Operating System. Trivy detects vulnerabilities of OS packages (Alpine, RHEL, CentOS, etc.) and application dependencies (Bundler, Composer, npm, yarn, etc.). Trivy is easy to use. Just install the binary and you're ready to scan. All you need to do for scanning is to specify a target such as an image name of the container.

Installation

Installing from the the Aqua Chart Repository.



helm repo add aquasecurity https://aquasecurity.github.io/helm-charts/ 
helm repo update 
helm search repo trivy 
helm install my-trivy aquasecurity/trivy

Installing the Chart.

To install the chart with the release name my-release:



helm install my-release .

The command deploys Trivy on the Kubernetes cluster in the default configuration. The Parameters section lists the parameters that can be configured during installation.

Know more about Trivvy installation here.

Use Cases

DevSecOps Integration: Trivvy seamlessly integrates into your CI/CD pipelines, identifying vulnerabilities early in the development process and enabling automated remediation.
Secure Container Deployment: It ensures only secure container images are deployed to production by scanning them before deployment and integrating with container registries for continuous scanning.
Ongoing Security Monitoring: Enables regular vulnerability scans and provides detailed reports, allowing for proactive security maintenance and tracking of remediation efforts.
Beyond Containers: Extend security assessments to operating systems, server configurations, and other infrastructure components.
Software Supply Chain Analysis: Analyze your entire software supply chain, from source to deployment, to identify and address vulnerabilities at every stage.

8. Cert-Manager

Cert Manager is an open-source tool designed to automate the management and provisioning of digital certificates in Kubernetes environments. It solves the challenge of handling TLS/SSL certificates for applications running on Kubernetes by simplifying the process of obtaining, renewing, and distributing certificates. Cert Manager enhances security and reduces operational complexity, ensuring that applications have valid and up-to-date certificates for secure communication.

It automates the lifecycle of your TLS certificates! No more manual renewal!

Installation

You don't require any tweaking of the cert-manager install parameters.

The default static configuration can be installed as follows:



kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.15.1/cert-manager.yaml

Checkout our blog to learn how to setup cert-manager using Devtron: https://devtron.ai/blog/kubernetes-ssl-certificate-automation-using-certmanager-part-1

Use Cases

Automated Certificate Acquisition & Renewal: Effortlessly obtain and renew certificates from providers like Let's Encrypt, eliminating manual effort and ensuring continuous security.
Secure Ingress Controllers: Automatically provision certificates for Ingress controllers, enabling secure HTTPS communication for services exposed through the Ingress.
Centralized Certificate Management: Manage all your certificates from a single point of control, simplifying issuance, renewal, and revocation.
Enhanced Application Security: Strengthen encryption and protect sensitive data by ensuring valid and up-to-date TLS/SSL certificates.
Streamlined Operations: Reduce operational overhead, minimize downtime, and ensure continuous application availability by automating certificate management.

9. Istio

Istio extends Kubernetes to establish a programmable, application-aware network. Working with both Kubernetes and traditional workloads, Istio brings standard, universal traffic management, telemetry, and security to complex deployments.

Installation

Check out the Istio docs for Getting Started and a complete walkthrough of Canary Deployment with Flagger and Istio on Devtron.

Use Cases

Simplify Microservice Communication:

Istio abstracts away complex networking concerns like service discovery, routing, and load balancing.
Developers can focus on business logic while Istio handles communication between microservices.

Enhance Security:

Implement consistent authentication, authorization, and encryption across all services using Istio.
It mitigates security risks by enforcing security policies throughout the mesh.

Traffic Management:

Istio enables A/B testing, canary deployments, and blue-green deployments.
You can control traffic routing, timeouts, and fault injection seamlessly.

Observability and Monitoring:

Monitor service behavior, track performance metrics, and troubleshoot issues with Istio.
It integrates with observability tools for better insights into your microservices.

10. KRR

Robusta KRR (Kubernetes Resource Recommender) is a CLI tool for optimizing resource allocation in Kubernetes clusters. It gathers pod usage data from Prometheus and recommends requests and limits for CPU and memory. This reduces costs and improves performance.

Installation

The installation is pretty straight-forward. You can install the binary directly from their releases. To install the CLI, depending upon your operating system, you can install the KRR cli and use it for optimising the resources. You can use brew for installing on mac:



brew tap robusta-dev/homebrew-krr

brew install krr

Use Cases

Cost Savings: Reduce cloud bills by recommending optimal resource requests, and eliminating over-provisioning.
Performance Boost: Improve application responsiveness by preventing resource contention and ensuring sufficient resources.
Data-Driven Insights: Gain insights into resource usage patterns for better planning and scaling decisions.
Automated Optimization: Integrate with CI/CD pipelines to automatically adjust resource allocation for continuous optimization.

11. Kyverno

Kyverno is a Kubernetes-native policy engine designed for Kubernetes platform engineering teams. It enables security, automation, compliance, and governance using policy-as-code. Kyverno can validate, mutate, generate, and cleanup configurations using Kubernetes admission controls, background scans, and source code repository scans in real time. Kyverno policies can be managed as Kubernetes resources and do not require learning a new language. Kyverno is designed to work nicely with tools you already use like kubectl, kustomize, and Git.

Installation

To install Kyverno with Helm, first add the Kyverno Helm repository.



helm repo add kyverno https://kyverno.github.io/kyverno/

Scan the new repository for charts.



helm repo update

Optionally, show all available chart versions for Kyverno.



helm search repo kyverno -l

Check the whole guide here.

And be sure to check out our blog on securing Kubernetes clusters with Kyverno policies.

Use Cases

Security: Enforce policies to prevent deployments with root privileges, restrict resource requests, control network access, and secure sensitive data.
Compliance: Implement auditing, labeling, and access control policies to meet regulatory requirements.
Best Practices: Automate resource validation, enforce naming conventions, and manage resource lifecycles.
Extend Kubernetes: Customize admission control and validate custom resources.

12. Opencost

OpenCost is a vendor-neutral open-source project for measuring and allocating cloud infrastructure and container costs. It’s built for Kubernetes cost monitoring to power real-time cost monitoring, showback, and chargeback. It is a sandbox project with the Cloud Native Computing Foundation (CNCF).

Installation

Check out the Installation guide to start monitoring and managing your spend in minutes. Additional documentation is available for configuring Prometheus and managing your OpenCost with Helm.

Use Cases

Identifying Costly Workloads: Identify specific pods or deployments consuming excessive resources and take steps to optimize their resource allocation.
Allocating Costs to Teams: Use OpenCost to generate detailed reports showing the cost incurred by each team or project using your Kubernetes cluster.
Right-sizing Resources: Optimize resource requests and limits for pods based on actual usage, reducing unnecessary resource allocation and saving costs.
Predictive Cost Management: Forecast future costs based on historical data and identify potential spikes in resource consumption to proactively adjust resource allocation or budget.

Conclusion

Kubernetes has revolutionized the way we build and deploy applications. However, its complexity can be daunting. The 12 tools we've explored provide a comprehensive toolkit for simplifying and optimizing your Kubernetes management.

The Coud-native landscape is overflowing with tools, each serving a specific purpose. Remember, there's no one-size-fits-all solution. Carefully evaluate your use case and choose the tools that best address your specific needs for a more efficient, secure, and cost-optimized Kubernetes experience.

By leveraging these powerful tools, you can unlock greater efficiency, reduce costs, enhance security, and ultimately, focus on delivering innovative applications faster. So, embrace these powerful tools and build a better infrastructure with confidence!

If you have any questions, or want to discuss any specific usecase, feel free to connect with us or ask them in our actively growing Discord Community.

Happy Deploying!

Reserving Minimum IPs In EKS Cluster

Devtron — Wed, 09 Oct 2024 13:02:00 +0000

The popularity of AWS Elastic Kubernetes Service (EKS) is consistently rising as a managed Kubernetes solution. From resource management, to networking, and implementing new requirements, EKS really comes with an easy user-friendly approach to overseeing all components. An abundance of good documentation and regular updates offered by the AWS community further enhance user experience, simplifying operations for end-users.

However, when it comes to the scalability of your workloads or Kubernetes cluster, challenges arise if proper planning was not undertaken during the initial phase of the cluster setup. A prominent issue arises in the management of IP addresses, a critical factor in scaling clusters. The insufficiency of available IP addresses within your subnets can precipitate an alarming shortage within your cluster. IP shortage can lead to operational challenges, impacting the deployment and functioning of applications. This article delves into this issue and its corresponding solution, providing an in-depth exploration of the matter.

Issues Arising from IP Shortages

If your EKS cluster facing an IP shortage issue then you would likely have come across the subsequent error message when attempting to deploy a new application or scale an existing one within the pod events:



#  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "82d95ef9391fdfff08b86bbf6b8c4b6568b4ee7bb81fce" network for pod "example-pod_namespace": network plugin cni failed to set up pod "example-pod_namespace" network: add cmd: failed to assign an IP address to container

Under such conditions, your pod will be stuck in the "ContainerCreating" state, unable to initiate until an IP address is assigned to it. Upon investigation of the cluster's private subnets, you'll likely discover that the specific subnet where the worker node is allocated and the pod is assigned exhibits an available IP address count of 0.

What could be a possible solution?

In order to understand the resolution for this issue, we have to first understand the functioning of the EKS cluster and how IP addresses get allocated to worker nodes and further to pods.

Understanding IP Allocation in AWS EKS Clusters

Within an AWS EKS cluster, IP addresses play a vital role in facilitating communication among worker nodes, services, and external entities. The management of IP addresses is primarily managed by the AWS-node daemonset as a default mechanism. This daemonset is responsible for the allocation of distinctive IP addresses to each worker node. It ensures that each worker node receives a unique IP address by requesting IPs from the Amazon VPC's IP address range associated with the cluster's subnet.

For the detailed information, check Amazon VPC CNI documentation

By default, upon the joining of a new worker node into the cluster, a specific number of IP addresses is affixed to the worker node, based on the associated network interfaces. For instance, consider the scenario of a compute-optimized instance such as c5a.2xlarge, having 2 network interfaces attached. In this instance, a total of 30 private IP addresses will be allocated from the corresponding subnet where this worker node has been allocated. Now it does not matter whether you have 5 pods running in this worker node or 15, these 30 IP addresses will be attached to this worker node which is obviously a wastage of lots of IP addresses. In the below screenshot, you can see the numbers of IPs attached to the existing node.

To view this option select worker node in EC2 instance console -> choose networking section.

To check each instance type default IP allocation count, refer to the AWS official documentation.

To check the available IP addresses on the subnet

Open VPC console -> choose subnet -> select subnets of the clusters

In the above snapshot, it is clear that in this subnet we have 1454 IP addresses available. So what if we ran out of IP addresses? Let’s see how it can be resolved if such a scenario exists.

Avoiding IP Shortages

There can be multiple ways to avoid IP shortages in your EKS clusters. Following are the different ways that can be helpful.

Adding ENI configuration in the cluster with different subnets. Check out this detailed documentation which talks about adding custom ENI configurations.
Update the existing cni-plugin to assign the minimum IP at the boot time of the new worker node.
Prefix Mode for Linux

In this blog post, we will proceed with the implementation of updating the existing cni-plugin as we don’t have to create extra objects. With this approach, we just have to update the existing cni-plugin which is the default cni-plugin provided by AWS EKS.

Updating existing CNI Plugin

To update the existing cni-plugin, we will add/configure 3 environment variables in the AWS-node daemonset. For detailed information, check out the official documentation which talks about the environment variables.

1. WARM_IP_TARGET

The number of Warm IP addresses to be maintained. A Warm IP is available on an actively attached ENI but has not been assigned to a Pod. In other words, the number of Warm IPs available is the number of IPs that may be assigned to a Pod without requiring an additional ENI.

For Example: Consider an instance with 1 ENI, each ENI supporting 20 IP addresses. WARM_IP_TARGET is set to 5. WARM_ENI_TARGET is set to 0. Only 1 ENI will be attached until a 16th IP address is needed. Then, the CNI will attach a second ENI, consuming 20 possible addresses from the subnet CIDR.

2. MINIMUM_IP_TARGET

The minimum number of IP addresses to be allocated at any time. This is commonly used to front-load the assignment of multiple ENIs at instance launch.

For Example: Consider a newly launched instance. It has 1 ENI and each ENI supports 10 IP addresses. MINIMUM_IP_TARGET is set to 100. The ENI immediately attaches 9 more ENIs for a total of 100 addresses. This happens regardless of any WARM_IP_TARGET or WARM_ENI_TARGET values.

3. WARM_ENI_TARGET

The number of Warm ENIs to be maintained. An ENI is “warm” when it is attached as a secondary ENI to a node, but it is not in use by any Pod. More specifically, no IP addresses of the ENI have been associated with a Pod.

For Example: Consider an instance with 2 ENIs, each ENI supporting 5 IP addresses. WARM_ENI_TARGET is set to 1. If exactly 5 IP addresses are associated with the instance, the CNI maintains 2 ENIs attached to the instance. The first ENI is in use, and all 5 possible IP addresses of this ENI are used. The second ENI is “warm” with all 5 IP addresses in the pool. If another Pod is launched on the instance, a 6th IP address will be needed. The CNI will assign this 6th Pod an IP address from the second ENI and from 5 IPs from the pool. The second ENI is now in use, and no longer in a “warm” status. The CNI will allocate a 3rd ENI to maintain at least 1 warm ENI.

You have the choice to implement this either by manually adding the required environment variables to the manifest or by using the patch command to configure the environment variables.

Note: Opting for the patch option is advisable, as it ensures that other elements within the manifest remain unaffected and unaltered.

Option 1: Adding the environment variables in manifest

Run the following command



kubectl edit daemonset aws-node -n kube-system

And add the environmental variables in the env section of the container i.e, spec.template.spec.containers.env as

env:

- name: WARM_IP_TARGET

value: "2"

- name: MINIMUM_IP_TARGET

value: "10"

Option 2: Adding env variables using patch



kubectl set env daemonset aws-node -n kube-system WARM_IP_TARGET=10 MINIMUM_IP_TARGET=2

After the implementation of this adjustment takes effect across each pod of the aws-node, you will observe that when a new node becomes part of the cluster, it will initiate with an allocation of only 12 IPs.

And it has reduced only 13 IPs from the subnet, one is for the node private IP and 12 are reserved for pods.

Conclusion

By reserving a specific number of warm IP addresses and ENIs, you ensure that each worker node has a minimum number of IPs available for pod assignment, reducing the risk of IP shortage. While the solution mentioned above offers advantages, it's important to be aware of a few tradeoffs:

If multiple pods are scheduled on a single node, exceeding the warm IP count, there might be a slight delay in the startup time for these pods.
In cases where multiple pods are scheduled on a single node, surpassing the warm IP count, and the subnet lacks any available IP addresses, the scheduling process will fail. In such scenarios, the remaining option involves utilizing extra ENI-configurations with distinct subnets.

IP shortage can pose challenges to the scalability and smooth functioning of AWS EKS clusters. By configuring the aws-node daemonset using the WARM_IP_TARGET, MINIMUM_IP_TARGET, and WARM_ENI_TARGET environment variables, you can effectively mitigate IP shortage concerns. This approach ensures that each worker node has a minimum number of IP addresses reserved for pod assignment while dynamically allocating additional IPs.

Feel free to connect with us on our Discord Community if you have any queries. We would be more than happy to help you.

Kubernetes Adoption: Key Challenges in Migrating to Kubernetes

Devtron — Mon, 07 Oct 2024 12:01:00 +0000

Many organizations have spent many years building and refining their software delivery infrastructure within non-kubernetes environments. They might run their infrastructure on cloud-hosted VMs or bare metal servers using a virtualizing tool such as Proxmox. While these methods are useful and get the job done, they have limitations beyond a certain scale.

To address this scaling issue, companies want to move their workloads over to a Kubernetes environment. Apart from providing improved scalability, Kubernetes also provides a lot of other benefits such as automation, efficiency, auto-healing, and flexibility. However, migrating the entire business workload to Kubernetes is a daunting task, and it has several challenges associated with it.

In 2024, it would found in the State of Production Kubernetes survey that nearly 75% of responders use Kubernetes for running their production applications, leaving only 25% of respondents using traditional infrastructure such as VMs for their production applications.

Before we explore the challenges of adopting Kubernetes let’s understand what the adoption journey might look like.

Kubernetes Adoption Journey

Kubernetes adoption is one of the most difficult adoption that most of the organizations would go through. The journey of adopting Kubernetes is no less than a roller coaster ride. Even before you start with creating your very first Kubernetes cluster, you would need to make sure your application is containerized. And to containerize your application, you need to make sure the application is ready for containerization. It takes most organizations months or years to gain complete Kubernetes maturity, depending on multiple factors such as the size and scale of existing applications, technical expertise, existing infrastructure, and more. Let’s take a look at the 4 different stages that every organization would go through to achieve Kubernetes Maturity.

Setting up Kubernetes: The first stage in any Kubernetes adoption journey is to create a Kubernetes cluster and get it ready for production deployments. You need to ensure that the cluster has the proper security and compliance before you start deploying your application to the cluster. All these tasks require Kubernetes expertise.
Migrating Workloads: Making your application complaint to the Kubernetes environment and onboarding your first application can be cumbersome. Comparatively, migrating all applications might involve repetitive tasks and is a time-consuming activity. A process has to be created that can help you onboard applications quickly, without worrying about the configurations and writing helm-charts or K8s-manifests.
Software Delivery Acceleration: There needs to be a proper process in place, which enables developers to accelerate their software delivery speed, while also ensuring that the proper compliance policies are being followed.
Day 2 Operations: Once your applications are all deployed onto Kubernetes you would want to make sure that they are stable. This means ensuring that they can be updated to newer versions using deployment patterns such as blue-green or canary, without any significant downtime. This also involves getting visibility into the environments and checking if there are any resource constraints, ensuring dynamic resource scaling to meet workloads, etc.

Challenges with Getting Started

Within the entire Kubernetes adoption journey, there are multiple different challenges that organizations face. Let’s look at some of these challenges, and understand why migrating to Kubernetes takes a significant amount of time.

Steep Learning Curve

Kubernetes brings a lot of flexibility to the table, which enables developers and operations teams to have faster release cycles, while also ensuring maximum reliability. However, these advantages come with a lot of added complexity and nuances. In the pre-Kubernetes era, we were used to handling a lot of VM-level abstraction. This laid down a different foundation, mental model, and building model for the infrastructure components.

Understanding these models in a Kubernetes context is a challenge. Rather, it takes a lot of time to understand these nuances well. Even after understanding the Kubernetes nuances, developers lack the confidence needed to deploy their applications onto Kubernetes. You don’t want to take your applications to production if you can’t control your applications to the best of your ability, as it might affect the SLAs set in place. Within the State of Production Kubernetes 2024 report by Spectro cloud, 77% of respondents said that Kubernetes complexities have inhibited their adoption journey.

Before Kubernetes, developers had a very simple world, where they needed to know only a few technologies and develop software to solve the business concerns. They didn’t need to worry about different infrastructure components, and if the application’s design worked well with the existing infrastructure. Learning about Kubernetes for their application takes time, and it’s a new burden for the developers.

Containerization

Kubernetes by its very nature, is designed to run workloads as microservices. When migrating to Kubernetes, one of the first challenges you will be facing is containerizing your workloads. If you’ve previously run a monolithic application, i.e. every component bundled into a single large application, you will need to break these components down into smaller individual self-contained applications i.e. microservices to obtain the maximum benefit by shifting to a Kubernetes cluster.

Kubernetes is a container orchestrator, which means that to run your workloads, each application will need to be containerized. Learning how to create, build, and use a container image adds to its learning curve. Moreover, depending on your application’s tech stack, the configurations needed for creating a container image would vary.

Challenges with Tool Integrations

The cloud-native ecoystem is nothing less than a pacific ocean. The deeper you go, the more you are lost. There are hundereds and thousands of tools built for specific use-cases and the biggest problem is to integrate and manage the tools-sprawls.

Broad Ecosystem

Kubernetes has an extremely huge and extensive ecosystem. If you look at the CNCF ecosystem, there are 100+ different tools, all solving different problems. To make your cluster production ready, it is necessary to integrate these tools within your clusters. However, just the huge number of solutions for every category can be overwhelming. You have to evaluate the tools and select the ones that fit your needs.

Even after you’ve evaluated the tools, and shortlisted a small number of tools that meet your requirements, there still is the question of how you are going to integrate these tools within your cluster. Some of the tools are straightforward to integrate, but many require a lot of additional configuration which adds a cognitive burden and introduces a learning curve for developers.

While having the choice to pick your tools offers quite a lot of flexibility, 48% of survey respondents state that it is very difficult to choose the right tools form the broad ecosystem.

Multi-cluster & Multi-cloud Strategy

A lot of different organizations try to adopt a multi-cluster or multi-cloud strategy for their workloads i.e. spreading their workloads across multiple Kubernetes clusters or multiple different cloud providers. There is also an increasing demand for adopting a Hybrid cloud strategy i.e. hosting some clusters on a public cloud such as AWS or GCP, and hosting other clusters in on-prem infrastructure. This can help with enhanced reliability for applications and ensures maximum uptime. However, managing a multi-cloud or hybrid workload is not easy.

One of the most prevalent challenges with a multi-cloud setup is a lack of visibility across all the clusters. There isn’t a uniform way in which you can look at the Kubernetes objects of all clusters in a single place. There is constant context switching between multiple clusters which can make it difficult to debug an application if needed or understand how certain components are related to each other.

Challenges with Securing Kubernetes Clusters

Every piece of software has a common shared struggle: being secure enough. Kubernetes is no different, and securing the cluster can become a huge challenge. Kubernetes has many features that help in security, but they can become overwhelming even for experienced K8s users. Let’s explore these challenges in detail.

Access Control Management

Managing the right level of access control for aspects of your Kubernetes cluster is a big challenge that many Kubernetes adopters face. For example, you might want to allow your developers to deploy applications in a staging environment, but not in a production environment. Out of the box, Kubernetes provides a lot of mechanisms for creating fine-grained access control for multiple different users.

The real challenge lies with managing access control and creating the right level of abstraction in real time without hindering the speed and agility of teams. Imagine if you accidentally gave super admin permissions to anyone within your organization. They would be able to do anything within the cluster, which might disrupt your services and lead to unhappy customers.

DevSecOps implementation

When running on Kubernetes, you want to ensure that you have robust DevSecOps practices in place. Before deploying your application to any environment, you should run some security scans on the application code and containers.

Evaluating tools, and integrating them within your CI/CD pipelines is quite a big challenge. Moreover, what if you want to set some governance policies based on the number of vulnerabilities found in the security scans? Setting these policies can also be a big challenge.

Conclusion

The adoption and migration journey from a traditional VM-based workload to a Kubernetes environment is quite long and filled with challenges at different levels. The learning curve is especially high because Kubernetes itself is a distributed system, and there are many different tools within the Kubernetes ecosystem that you will need to learn about. Even after getting past the high learning curve, there are still many challenges with security, different tool stacks, setting up monitoring, etc, and then managing all of these different aspects within the cluster. This adoption journey can typically take anything from a few months, to even a few years depending on the scale of your organization.

Devtron is a modern Kubernetes dashboard that simplifies a lot of the challenges with Kubernetes. With the help of Devtron, you can reduce your adoption time from a couple of months to just a couple of weeks. It helps reduce a lot of the unnecessary complexities in Kubernetes so that you can focus on developing your applications, and not worry about the operations. To learn more about how Devtron addresses all the challenges mentioned above, check out the second part of this article.

Setting up custom DNS routing on EKS Cluster

Devtron — Wed, 02 Oct 2024 10:52:00 +0000

Our one of the third party API URL was failing to resolve, so we figured out the solution to route through Google Public DNS, thus changing the routing of a particular domain from EKS Default DNS ( 10.100.0.10 ) to resolve using Google Public DNS.We used 8.8.8.8, the primary DNS server for Google DNS, in order to function it correctly.

Configure Conditional Forwarder with CoreDNS in Amazon EKS cluster

What is CoreDNS?

CoreDNS is a DNS server that is modular and pluggable, and each plugin adds new functionality to CoreDNS. This can be configured by maintaining a Corefile, which is the CoreDNS configuration file.
As a cluster administrator, you can modify the ConfigMap for the CoreDNS Corefile to change how DNS service discovery behaves for that cluster.
CoreDNS uses negative caching whereas kube-dns does not (this means CoreDNS can cache failed DNS queries as well as successful ones, which overall should equal better speed in name resolution).

You can use CoreDNS to configure conditional forwarding for DNS queries that are sent to the domains resolved by a customized DNS server(like Google DNS Server).

How Amazon EKS uses CoreDNS?

Pods running inside the Amazon EKS cluster use the CoreDNS service’s cluster IP as the default name server for querying internal and external DNS records.

You can follow the mentioned steps to modify the CoreDNS ConfigMap and add the conditional forwarder configuration.

1. Run the following command:

$ kubectl -n kube-system edit configmap coredns

Output of the command should be:

apiVersion: v1 
kind: ConfigMap 
metadata: 
  annotations: 
  labels:
    eks.amazonaws.com/component: coredns 
    k8s-app: kube-dns 
  name: coredns 
  namespace: kube-system 
data: Corefile: | 
        .:53 { 
            errors 
            health 
            kubernetes cluster.local in-addr.arpa ip6.arpa { 
              pods insecure 
              upstream 
              fallthrough in-addr.arpa ip6.arpa 
            } 
           prometheus :9153 
           proxy . /etc/resolv.conf 
           cache 30 
           loop 
           reload
           loadbalance 
       } 
       domain-name:53 { 
           errors
           cache 30 
           forward . custom-dns-server 
           reload 
     }

Note: We have customized the above configMap with the domain-name “plapi.ecomexpress.in. Replace it with your domain name.

The custom-DNS-server IP address for Google DNS is used, that is (8.8.8.8). Replace the custom DNS server IP address with your custom DNS server IP address.

2.The final CoreDNS ConfigMap will look like:

apiVersion: v1
data:
    Corefile: |
         .:53 {
               errors
               health
               kubernetes cluster.local in-addr.arpa ip6.arpa {    
                   pods insecure
                   upstream
                   fallthrough in-addr.arpa ip6.arpa
                 }
                 prometheus :9153
                 forward . /etc/resolv.conf
                 cache 30
                 loop
                 reload
                 loadbalance
}
plapi.ecomexpress.in:53 {
       errors
       cache 30
       forward . 8.8.8.8
       reload
}
kind: ConfigMap

3. To verify that domain-name resolution works, run the following command:

prod@ip-192-168-X-XXX:/home/devtron$ kubectl exec busybox -- nslookup domain-name.in

Note: Replace the domain-name with your domain name.

The output before updating custom route for CoreDNS:

prod@ip-192-168-X-XXX:/home/devtron$ kubectl exec busybox -- nslookup plapi.ecomexpress.in

Server:    10.100.0.10
Address 1: 10.100.0.10 kube-dns.kube-system.svc.cluster.local
Name:      plapi.ecomexpress.in
Address 1: 172.20.92.37 ip-172-20-92-37.ap-south-1.compute.internal
Address 2: 172.20.54.52 ip-172-20-54-52.ap-south-1.compute.internal

The output after updating custom route for CoreDNS:

prod@ip-192-168-X-XXX:/home/devtron$ kubectl exec busybox -- nslookup plapi.ecomexpress.in

Server:    10.100.0.10
Address 1: 10.100.0.10 kube-dns.kube-system.svc.cluster.local
Name:      plapi.ecomexpress.in
Address 1: 35.154.40.19 ec2-35-154-40-19.ap-south-1.compute.amazonaws.com
Address 2: 3.6.218.14 ec2-3-6-218-14.ap-south-1.compute.amazonaws.com

Creating a Production grade EKS Cluster using EKSCTL

Devtron — Mon, 30 Sep 2024 10:51:00 +0000

What Is EKSCTL?

EKSCTL almost automates much of our experience of creating EKS Cluster. EKSCTL is written in Go and makes use of AWS service, CloudFormation. It is the official CLI for Amazon EKS. The current version of eksctl allows you to create a number of clusters, list those, and delete them as well.

Amazon Production Grade EKS Cluster with One Command:

When we look at creating a Production grade EKS Cluster, we can create an EKS Cluster with the following command: eksctl create cluster

When you run the above command, following things happen:

Sets up the AWS Identity and Access Management(IAM ) Role for the master plane to connect to EKS.
Creates the Amazon VPC architecture, and the master control plane.
Brings up instances, and deploys the ConfigMap so nodes can join the cluster.
Provides access to the cluster with a pre-defined kubeconfig file.

Create Production Grade EKS CLuster: Using Config Files

You can create Production Grade EKS Cluster using the Config File. Following are the steps:

First, attach the following AWS Managed Policies for a role / user / group required for creating an EKS Cluster using EKSCTL

AmazonEC2FullAccess
IAMFullAccess
AmazonVPCFullAccess
AWSCloudFormationFullAccess

Second, Create a Cluster.yaml File

Properties:

OnDemandPercentageBaseCapacity: The minimum amount of the Auto Scaling group’s capacity that must be fulfilled by On-Demand Instances. The default value is 0, in this On-Demand Instances are launched as a percentage of the Auto Scaling group’s desired capacity as per onDemandPercentageAboveBaseCapacity setting.

OnDemandPercentageAboveBaseCapacity: Controls the percentages of On-Demand Instances and Spot Instances for your additional capacity beyond onDemandPercentageBaseCapacity. The range is 0–100. The default value is 100. Here, this property set to 50, the percentages are 50% for your additional capacity above base capacity.

vpc and subnets: If you don’t define these two properties, then AWS will automatically create vpc and subnets and assign them with their respective id’s.

attachPolicyARNs: Attaches the specified managed policy to the specified IAM role. Here, you will have to define custom policies along with managed policies because policies are explicitly defined, if you decide to leave it blank AWS will implicitly attach it’s own policies for creating an EKS Cluster.

Next, run this command to create EKS cluster using your yaml file: eksctl create cluster -f cluster.yaml

That’s it ! Your Production Grade EKS CLuster is ready. For eksctl documention, check the following link: https://eksctl.io/introduction/#getting-started/

To continue learning more about EKS, read this blog post on how to set up custom DNS routing on an EKS cluster.

AWS Cost Optimization Parameters and Metrics: Part-2

Devtron — Fri, 27 Sep 2024 12:51:00 +0000

Are you trying to learn more about AWS cloud cost management? Is your monthly bill of AWS, surging and you're perplexed, if you are using all the resources that you have paid for? Here, we are back with an extended version of our previous blog on AWS Cost Optimization.

It is a well-known fact that moving to Amazon Web Services(AWS) can provide a huge number of benefits in terms of agility, responsiveness, much-simplified operations, and improved innovation. However, there is an assumption that migrating to the public cloud will lead to cost savings. Though in reality, AWS cost optimization is tougher than you can think of.

To optimize your costs, you need to know exactly what are your organization needs, how and when they are used, while adapting to the increased demands of agile and fast pacing technology.

In this blog, we will cover 4 indispensable pillars of cost optimization

Pillar 1: Cost effective resources

1. Autoscaling

Automating the deployment of applications requires software tools and frameworks, in addition to proper infrastructure (with enough resources, such as servers and services).

You can operate the provisioning of test environments using manual based efforts such as AWS’s APIs or Command Line Interface(CLI) tools. However, the manual intervention leads to lower productivity from the teams. Hence, there is an absolute need to automate infrastructure processes (like provisioning test environments) which will improve productivity and efficiency. Some of the ways in which you can automate processes are,
1. Provisioning EC2 instances via Amazon Machine Images (AMI): AMI encapsulates OS and other software/configuration files. When an instance starts, all the applications come pre-loaded from the AMI. AMIs enable the launch of standardized instances across multiple regions, by allowing the copying of AMIs from one region to another.
2. Deploying platforms using Amazon Elastic Beanstalk: With AWS Elastic Beanstalk, you can easily deploy and scale web applications and services developed with Node.js, Python etc, on familiar servers such as Apache, Nginx, and IIS. You just need to upload your code and Elastic Beanstalk will automatically handle the deployment, capacity provisioning, load balancing, auto-scaling, and application health monitoring. At the same time, you retain full control over all the cloud resources powering your application.

2. Right sizing of instances

Based on an analysis of OS instances, in North America, it was found that about 84% of instances were not correctly sized. It was estimated that, right-sizing of instances could lead to a cost reduction by 36% ( USD 55 million). The right-sizing could be achieved by porting them to optimally sized AWS resources.

You can achieve right-sizing by making sure:

You use the correct granularity for the time period of analysis that is required to cover any system cycles.For e.g., if a two week analysis is performed, you might be overlooking a monthly cycle of high utilization, which could lead to under-provisioning.
Right-sizing is an iterative process, that gets triggered by changes in usage patterns and external factors like AWS price drops or new AWS resource types.

3. Purchasing options to save cost

Amazon EC2 provides a number of purchasing models for instances. Using the Reserved Instance purchasing model, can help you save up to 75% over On-Demand capacity. Spot Instances are another phenomenal way to save money for non-stateful workloads.

Reserved instances

Reserved Instance is a 1 or 3 year commitment towards purchasing a reservation of capacity. In this case, you will significantly pay a lower hourly rate. If you use reserved instances, they enable up to 75% savings over on-demand capacity. Moreover, you get a chance of selling the unused Reserved Instances.

Spot instances

Spot Instances provide a way for saving, by allowing you to use spare compute capacity at a significantly lower cost than on Demand instances(up to 90%). You can also use spot instances to increase your computing scale and throughput for the same budget.

Spot instances can be used when you need large computing capacity such as for batch processing, scientific research, financial analysis, testing, and when you are not concerned about any interruptions, provided you have ways to deal with such interruptions.

4. Use of correct AWS S3 storage cclass

Storage in AWS is cheap and finite, but that’s not a good reason to keep your organization’s data there forever.

It is necessary, to clean the data from S3 time-to-time, so as to optimize the storage costs. Here, are some of the storage methods, that can be used:

Amazon S3 Standard Infrequent Access (S3-IA): You can use this type of storage for storing data that is accessed less frequently. The data can be retrieved rapidly whenever needed. The major drawback is, you are charged a retrieval fee of $0.01 per GB and uses 128 blocks to store data. If you have smaller objects, then it will be more expensive than the standard storage.
Amazon S3 Glacier: It can be used for archives, where a portion of the data might be required to be retrieved within minutes. The data stored in S3 Glacier has a minimum storage duration of 90 days and can be retrieved within 1-5 minutes.
Delete Policy: For those files that you think might not be required anymore, set up a delete policy.

For pricing information on Amazon S3, click on Amazon S3 Pricing

5. Geographic selection

AWS Cloud infrastructure is built around Regions and Availability Zones. As of March 3, 2020, AWS has 16 public regions and 2 non-public regions. Each region operates within the local market conditions, and resource pricing can be different for each region. You have to choose a specific region to architect your solution, so that you can run at the lowest price globally.

Let's consider a sample workload - 5 c5.large, 20 GB gp2 EBS storage each, 1 ELB, 5.1 TB data processed. 1 ELB sends traffic to 5 c5.large instances running Amazon Linux in the same availability zone. Each instance has 20 GB of EBS SSD storage, and each instance receives 100 GB/month from the ELB and sends 1 TB/month back to the ELB. Therefore, the ELB processes 5.1 TB/month. If we consider this workload, there is a substantial cost difference in AWS pricing across different Regions. It costs 52% more to deploy this infrastructure in a location in South America, compared to a location in North America.

While you choose a region to architect your solution based on the geographic location that minimizes your cost, it’s a best practice to place computing resources closer to users, so as to provide lower latency and strong data sovereignty.

Pillar 2: Matching supply and demand

You can deliver services at a low cost, when the infrastructure is optimized. This can be done using the following methods:

1. Demand-based approach

This approach leverages Elasticity(ability to scale up or scale down, managing capacity and provisioning resources as demand changes) of AWS Cloud. AWS provides APIs or Services for the dynamic allocation of cloud resources to your application / solution. As per AWS best practices, you should use AWS auto-scaling. It is a service that makes scaling simple with recommendations that allows you to optimize performance and cost.

2. Buffer-based approach

A buffer in AWS will allow your applications to communicate with each other when they are running at different rates over time. This approach involves decoupling components of a cloud application and creating a queue that accepts messages. A buffer will queue the request, until the resources are available.
This approach is suitable if you have workloads that are not predictable or time-sensitive. Some of the key AWS services that enable this approach are Amazon SQS and Amazon Kinesis.

If you have a workload that generates write load, and need not be processed immediately, you can use the buffer to smooth out demands on resources.

3. Time-based approach

This approach involves aligning resource capacity to demand, that is predictable over specified time periods. If you know when resources are going to be required, you can time your system to make the right resources available at the right time. You can implement time-based resource allocation by timing your auto scaling. However, while using auto scaling for this approach, you need to be careful with the following:

Load based auto scaling is not always appropriate in every situation. For e.g., cloud deployments for small startups which have less than 50 instances and witness unusual patterns. In such cases, the close matching of demand and supply may not be optimal.
Auto scaling can add a new instance in 5 mins, and take another 3-5 mins to start a new instance. Due to this mismatch of enough instances to handle the load, over-loading of existing instances can occur. This in turn, slows down the health check, and ELB removes the instance, which can worsen the situation.

Pillar 3: Expenditure monitoring

One of the most crucial drivers for effective decision making in the organization, is having crystal clear view of AWS Resource Metrics. AWS recommends the following approaches to achieve expenditure monitoring:

1. Stakeholders

It is a good practice to have necessary stakeholders involved in the expenditure awareness discussions, as it produces better outcomes. It is recommended to involve financial stakeholders such as CFOs, Business Unit Owners, and any Third parties that might be directly involved in resource expenditure.

This will bring any hidden costs into forefront, which would provide opportunities for cost optimization, and would make sure that costs will be correctly allocated to the right business unit.

2. Reserved instance reporting

Reserved Utilization Report and RI Coverage Report are the key tools that help you in analyzing the costs. These reports visualize the percentage of running instance hours, that are covered by reserved instances. The reports may visualize the content in an aggregate way or in a detailed way (by account, instance type, region, availability zone, tags, platform etc).

What is Reserved instance utilization report?

It allows you to visualize RI utilization (% of purchased RI hours consumed by the instances during a period of time) and shows how much savings are accrued, due to the usage of reserved instances.

What is Reserved instance coverage report?

It allows you to discover how much of the overall instance usage is covered by RIs, so you can make informed decisions about when to modify or purchase RIs to ensure maximum coverage.

Pillar 4: Optimizing over time

1. Establishing cost optimization function

You can establish a function within your organization.
This function will be performed by an existing team such as Cloud COE, or you can even create a new team of key stakeholders from the appropriate BUs in the organization.
This function will coordinate and manage all aspects of cost optimization, from your technical teams, to your people and all the processes.

2. Monitor, track and analyze your service usage

AWS recommends establishing strong goals and metrics for the organization to measure itself against. These goals should include costs, but should also surface the business output of your systems to quantify the impact of your improvements.
AWS suggests using tools like AWS Trusted Advisor and Amazon Cloud Watch to monitor your usage and handle the workloads accordingly.
You can use Consolidated Billing, if you have multiple AWS accounts. This service has no additional charges. It is a very helpful method, which enables you to see a combined view of all the charges across all of your AWS accounts.

Now that you know the four pillars, share your cost optimization experience with us in the comments below. Connect with us on our community Discord server

Adopting kubernetes also plays a vital role in optimizing cost if implemented with a right approach. In addition to cost optimization, it also helps to reduce carbon footprint released by the organization.

How to deploy Kubernetes Secrets with AWS Secrets Manager

Devtron — Mon, 23 Sep 2024 10:50:00 +0000

In Kubernetes, external secrets refer to managing sensitive information, such as API keys, database passwords, or other credentials, outside of the Kubernetes cluster and then securely injecting them into the cluster when needed.

This approach is crucial for several reasons:

Security: Storing sensitive information directly in Kubernetes manifests or configuration files is a security risk. External secret helps to reduce the risk by keeping the data separate from the application code and configuration which reduces the chance of accidental exposure or misuse of credentials.
Separation of Concerns: Externalizing secrets allows us to separate application code and configuration which is a major concern. It allows developers to focus on writing code without worrying about handling sensitive data. The operations team can manage secrets separately by applying the best practices without impacting application logic.
Centralized Management: External secrets facilitate centralized management of sensitive information. This means secrets can be stored, rotated, and audited in a centralized system outside the Kubernetes cluster. Centralized management simplifies the task of maintaining and updating credentials without the need to modify application code or configurations.
Dynamic Secrets: Some external secret management systems support dynamic secrets, which are credentials that are generated on-demand and have a limited lifespan. This enhances security by minimizing the exposure window for sensitive information. Kubernetes workloads can request dynamic secrets as needed, reducing the risk of unauthorized access.
Integration with Secret Management Tools: External secrets can be integrated with different secret management tools like HashiCorp Vault, AWS Secrets Manager, etc. These tools provide advanced security and features such as encryption, access controls, and audit trails.
Compliance: Many organizations need to adhere to specific compliance standards that mandate secure handling of sensitive information. Externalizing secrets and leveraging external secret management tools can help meet these compliance requirements.

Some common third-party secrets management tools include:

HashiCorp Vault: A tool for managing secrets and protecting sensitive data.
AWS Secrets Manager: A service for managing sensitive information used by AWS services and applications.
Azure Key Vault: A cloud service for securely storing and managing secrets.
Google Secret Manager: Managed service provided by Google for storing secrets and confidential data.

In this blog, we will dive into AWS Secrets Manager and deploy secrets in Kubernetes using External Secrets Operator.

Deploy secrets from AWS Secret Manager in Kubernetes using Devtron

Prerequisite: Create Secrets in AWS Secret Manager

To add secrets in the AWS secret manager, do the following steps:

Go to the AWS secret manager console
Click Store a new secret
Add and save your secret

Deploy secrets in Kubernetes

Step-1: Deploy External Secret Operator (ESO)

External Secrets Operator is a Kubernetes operator that integrates external secret management systems like AWS Secrets Manager, HashiCorp Vault, Google Secrets Manager, Azure Key Vault, and many more. The operator reads information from external APIs and automatically injects the values into a Kubernetes Secret.

Helm chart link: https://charts.external-secrets.io

Step-2: Deploy the aws auth creds in secret name aws-secret-auth

Create a Kubernetes secret in the namespace in which the application is to be deployed using base64 encoded AWS access-key and secret-access-key. You can use Devtron's generic-helm-chart for it.

Step-3: Create a secret for your application

Go to the App Configuration >> Secrets and click on Add Secret

Select AWS Secrets Manager under External Secret Operator (ESO) from the drop-down. You can also see all other options available, and if there are requirement of any other third-party secret which is not available as of now, that can also be supported as long as it is supported by ESO.

AWS SecretManager
Google Secret Manager
Hashi Corp Vault

Step-4: Configure the secret

To configure the external secret that will be fetched from the AWS Secret Manager for your application, you will need to provide specific details using the following key-value pairs:

If you don't want to authenticate using access-key and secret-access-key, attach the SecretManagerReadWrite policy to your node and then the system should automatically fetch the secrets from the secret manager and deploy it into your Kubernetes cluster.

Once the policies are attached, you can change the SecretStore configs as specified below.

Feel free to check out the official documentation of External Secrets Operator to find out how authentication can be done. https://external-secrets.io/latest/provider/aws-secrets-manager/#aws-authentication

Interested in mastering the use of HashiCorp Vault within Kubernetes? Dive into our insightful blog to grasp how to deploy the Hashicorp Vault and integrate the fetched secrets into your Kubernetes applications.

Conclusion

As we wrap up, it's clear that unlocking the vault and harnessing the power of external secrets in Kubernetes is not just about securing your applications—it's about future-proofing your infrastructure, enabling innovation, and staying ahead of emerging threats.

Our exploration across tools like AWS Secrets Manager and HashiCorp Vault aims to bring order to the often turbulent world of secret management. Investing in these secure secret management practices today is safeguarding your assets while fostering an environment prepared for consistent growth and success.

If you have any questions feel free to reach out to us. Our thriving Discord Community is just a click away!

AWS EKS vs KOPS: Choosing the Right Kubernetes Solution

Devtron — Fri, 20 Sep 2024 10:50:00 +0000

What is AWS EKS?

Amazon Web Services (AWS) has revolutionized the way businesses handle their workloads with its Elastic Kubernetes Service (EKS). As a managed service, AWSEKS simplifies the process of deploying, managing, and scaling containerized applications. This is especially beneficial for DevOps teams, as it integrates seamlessly into their workflows, enhancing functionality and efficiency. The service provides a robust orchestration system, allowing teams to automate the deployment and scaling of applications. AWSEKS is designed to be highly available and scalable, ensuring that workloads are efficiently managed without sacrificing performance. By leveraging the power of Kubernetes, a widely adopted container-orchestration system, EKS offers an optimal solution for handling complex applications on the cloud, making it a cornerstone of modern cloud infrastructure. Even though EKS fully manages the Kubernetes control plane, the disadvantage is that you are unable to make changes to this control plane and it doesn’t allow you to have access to master nodes.

What is KOPS?

Kops is an acronym for “Kubernetes Operation”, it offers CLI tools that make creating and managing Kubernetes easy. It came before AWS EKS in 2016. It gives you complete control over the Kubernetes Environment. Using Kops, you can simplify the Kubernetes cluster set up since it gives you access to set up both master and worker nodes.

1. Kubernetes Cluster Setup

This is the foremost point to consider when evaluating Kubernetes solutions on AWS is how difficult it is to set up a working Kubernetes cluster.

Setting up a Kubernetes Cluster with EKS

Setting up a cluster with EKS is fairly complicated and has some prerequisites. Since EKS does not actually create worker nodes automatically so you must manage that process.

Kubernetes on AWS

You must have set up AWS CLI and AWS-IAM-authenticator as a prerequisite.
To manage the process of setting up worker nodes since EKS won’t do it for you, this can be done using Cloud Formation templates or EKSCTL, Check Creating Production clusters using EKSCTL.
To manage the process of setting up worker nodes, you can also use Terraform which allows you to set up a VPC and subnets set up to use the EKS module.

Setting up a Kubernetes Cluster with KOPS

Setting up a Kubernetes Cluster with KOPS is simpler than EKS. Since Kops manage most of the AWS resources required to run a Kubernetes cluster.

It can create and run your Kubernetes cluster with kops create cluster command.
It can manage most of the AWS Resources that you need to set up a Kubernetes cluster.
It will work with either a new or existing VPC.
Kops also allows you to generate terraform configuration for the AWS resources instead of directly creating them.

2. Kubernetes Cluster Management

After setting up a Kubernetes cluster, you must also consider what it is like to scale nodes, perform cluster upgrades, and integrate with other services.

Managing a Kubernetes Cluster with EKS

Managing a cluster using EKS is easier compared to Kops, the extra effort that you required to set up EKS using either CloudFormation or Terraform just pays off when it comes to cluster maintenance.

With EKS, you don’t have to bring your entire cluster down for upgrades and updates.
EKS is much more scalable, because of it’s highly available and fully managed control plane, you don’t have to worry if the cluster gets larger.
EKS also gives you a detailed version for the internal pods management. You can easily know about how pods communicate with each other, with the VPC and other AWS Services.
EKS allows you to add worker nodes by increasing the size of your AutoScaling Group.
EKS also allows you to replace worker nodes using kubectl drain and then terminating EC2 instance and do most upgrades without disturbing the cluster.

Managing a Kubernetes Cluster with KOPS

Though it is really easy to create a Kubernetes cluster on kops but it’s a real pain when it comes to manage the cluster. This can be observed by the following points:

You have to do a lot of work to upgrade and replace master nodes for the newer version of Kubernetes.
It uses private networking for pods by default.
Kops is a little further behind on Kubernetes versions than the EKS team, which is kind of an added liability.

3. Configuration and Access

One biggest difference between EKS and kops is in how control and access are handled in your Kubernetes Cluster.

Configuration and Access using EKS

With EKS, managing the master node, configuring cloud environments and tasks as such are handled by Amazon thus leaving you with absolutely no control over it. It might suit the developers but not for the server administrators who appreciate more control over the entire environment.

Configuration and Access using KOPS

Kops let you configure cloud environments, include configurations the way you like them. It thus increases efficiency and making you responsible for making sure the cluster is configured correctly by giving you complete control over the cloud environment. When you choose kops, you also have to make sure that you keep the master nodes working properly and always up to date. It is a server administrator’s favorite since they appreciate having complete control over the entire cloud environment.

4. Cost

It is another one of the biggest difference while you choose between EKS and Kops. After all, reducing the existing infrastructure cost can be an achievement for any organization.

Cost of running AWS EKS Kubernetes Cluster

Cost and Pricing

Kubernetes control plane can be used for a flat usage fee of $0.20 per hour or ~$145 per month for each EKS cluster that you create, depending upon the cluster size.
You cannot use spot instances for the cluster thus cost is more.
You at least need to keep 3 master nodes up for higher availability of AWS EKS Kubernetes Cluster.
If you wish to use the Cluster for Production, EKS will be cheaper but for test and Dev Environments EKS will be costlier than kops.
When you have large production clusters with high load, it is profitable to use EKS.

Cost of running KOPS Kubernetes Cluster

It is an open-source tool and is completely free to use but you are responsible for maintaining the infrastructure created by kops to manage your Kubernetes cluster.
You also have to manage the master nodes in KOPS, which adds additional cost.
It is cheaper to use KOPS when you are using a small or temporary cluster because then you don’t have a huge load on the master.
If you use 3 x t2.medium (or t3, they are cheaper) instance types as master nodes (~$100/month).
For Dev and Staging environments, you can reduce the running cost of the cluster by keeping single master nodes per cluster, which can further be reduced if you use spot instances for master nodes.

You can significantly reduce your Dev/Staging cluster costs with KOPS by keeping single master node or by running master nodes as spot instances or both. However, for Production grade clusters, that require high availability configuration, EKS is always a cheaper option instead of running 3 Master nodes (atleast t3/m4/c4/r4.large instances) ondemand.

5. Kubernetes Security

Security should be a top concern for every Kubernetes administrator. As the Kubernetes ecosystem matures, more vulnerabilities will be found and this should not be ignored.

Security of EKS Cluster

Security Kubernetes Cluster

You will get the benefit of securing your Kubernetes cluster from AWS on a platform and if you have some issues with the control plane that are also resolved by AWS support for EKS.
Your cluster has an additional layer of protection, since your AWS Account doesn’t have root access to your master nodes.
You can also set up EKS with encrypted root volumes and private networking.
Also, EKS clusters are set up with limited administrator access via IAM.

Security of KOPS Cluster

The security of cluster while using kops is entirely up to you, you can further increase the security since you have complete control over the master nodes.
Kops clusters does benefit from Amazon Shared Responsibility Model but without extra benefits of security expertise or support.
Private networking, encrypted root volumes, and security group controls are already included in most of the kops cluster.

Conclusion

AWS EKS gives you easy and hassle-free management of Kubernetes control plane, allows you to easily upgrade or update your Kubernetes cluster. It makes cluster maintenance easier and comes bundled with AWS Security and AWS support for your cluster.

Whereas, KOPS gives you a better command of your Kubernetes control plane to you but on the other hand is a little complex to manage and upgrade when compared with Amazon EKS. Though the KOPS community offers tutorials and support for using the tool.

Regardless of which solution you end up with (based on your unique use case and requirements) you’ll want to consider adding Devtron immediately to every K8s cluster you create. Using Devtron will abstract away the complexity of Kubernetes from your developers so they can efficiently build and deploy their applications across the clusters you manage.

3-Minute Strategy to Efficient AWS S3 Storage Management

Devtron — Wed, 18 Sep 2024 12:49:00 +0000

In this quick read, let's understand about AWS S3 storage bucket retention policy and its benefits.

Lifecycle policy

A lifecycle policy is used to move objects in your bucket, automatically from one storage class to another.

S3 storage classes

S3 Standard : S3 Standard offers high durability, availability, and performance object storage for frequently accessed data.
S3 Intelligent-Tiering: The S3 Intelligent-Tiering storage class is designed to optimise costs by automatically moving data to the most cost-effective access tier, without performance impact or operational overhead.
S3 Standard-IA : S3 Standard-IA is for data that is accessed less frequently, but requires rapid access when needed.
S3 One Zone-IA : S3 One Zone-IA is for data that is accessed less frequently, but requires rapid access when needed. S3 One Zone-IA stores data in a single Availability Zone and costs 20% less than S3 Standard-IA.
S3 Glacier : S3 Glacier is a secure, durable, and low-cost storage class for data archiving.

Benefits of retention policies

Cost Optimisation: Rules / Policies will help manage your storage costs by controlling the lifecycle of your objects. Create a lifecycle rule to automatically transition your objects to Standard-IA storage class, archive them to Glacier storage class, and remove them after a specified time period.

Logs Lifecycle Automation: You upload logs to S3 bucket, and you need those logs for specific period of time. For e.g., one month or three months. After that, you may want to archive or delete them.

Quick Access: To begin with, you place your files in a frequently accessed storage type. But after sometimes, you realise that the files will not be accessed frequently, and you want to archive them for a specific period of time. You might also decide to delete them later.

Steps to setup the lifecycle / retention of objects

Let's proceed with the assumption that you are already sending your application logs to S3. We will focus on configuring the lifecycle of the logs. In this case, let's move the logs from Standard to One Zone-IA class storage.

Log on to AWS console -> click on Services -> select S3
Click on the bucket you have created, now click on the*Management* section, and select the Lifecycle****option and clickon Add lifecycle rule.
After clicking ‘Add lifecycle rule’, a window will appear

[add rule name as per your requirement and choose rule scope]

Prefixes and Tags: If your bucket has folders and tags, then you can add their names here, in this prefix and tag fields. This will help you to differentiate between different folder’s lifecycle process. If you don’t have any sub folders, then select your whole bucket (in my case, I have selected whole bucket).
Now click on Next
In the transitioning step, we will add our lifecycle rules.
Select your bucket version (if you already enabled versioning while creating a bucket and you want to transition your logs according to versions, select the previous version, if not select current version)
Now add the transition, and enter your rules. For e.g., in how many days, youwant to move your objects to different storage classes.
Here, I am moving the files from Standard to OneZone-IA storage after 180 days and to the Glacier after 365 days of creation.
In the next step, set the expiration of the objects. In how many days, after the object’s creation date, an object should get automatically deleted. Here, I am deleting all the files after 366 days of creation.
As we upload a large number of files in S3, many times, the files fail to upload. In such cases, we can delete the incomplete files in 10 days.
Click on Next, and review the options you have entered/selected. Once reviewed, click on Create. Your rule will be created and attached to the bucket. You can also see the created life cycle policy under the management section in your bucket.

With few steps, you have learnt to configure your AWS S3 storage in an optimal and cost-effective manner.

Also check out this blog post about using spot to achieve cost savings on Kubernetes.

Unlocking the Power of Ephemeral Environments with Devtron

Devtron — Mon, 16 Sep 2024 10:48:00 +0000

In the world of software development, ephemeral environments are temporary setups that serve specific purposes, such as testing or staging new features. These environments are short-lived, designed to exist only for the duration of their use case—like testing a feature branch—before being dismantled.

Ephemeral environments contrast with traditional static environments, which are permanent and can lead to inefficiencies, especially when underutilized. They offer a dynamic approach, allowing developers to create an isolated environment on demand without affecting the main codebase or other ongoing development activities.

The Technical and Business Value of Ephemeral Environments

Ephemeral environments provide significant advantages in different sectors as mentioned below:

Cost Efficiency: By creating environments only when needed and tearing them down afterward, organizations avoid the cost of maintaining idle resources. This is particularly beneficial for companies where lower-end environments such as dev-env, and non-prod can cost up to five times more than production environments.
Agility and Speed: Developers can quickly spin up environments to test new features or bug fixes without waiting for access to a shared environment. This agility accelerates development cycles and time-to-market.
Risk Reduction: Testing in isolated environments ensures that unstable code does not affect the rest of the system, reducing the risk of bugs in production.

Is an Ephemeral Environment Right for You?

Deciding whether ephemeral environments are suitable for your organization involves considering your development needs and organizational goals. Key questions include:

Do your development teams frequently need isolated environments for testing?
Are you looking to optimize costs associated with non-production environments?
Is there a need to increase deployment speed and reduce risk in production?

If you answered "yes" to any of these, ephemeral environments could be highly beneficial for you.

Traditional Approach to Ephemeral Environment

Ephemeral environment as mentioned above are the short lived environments, created and destroyed once the task is completed. We can create our scripts maybe in Terraform or Ansible or in python/shell to spin up a complete new environment that can be your VM Machines or Kubernetes clusters. Even though the automation can be achieved, there are few disadvantages associated with this approach, that include:

Delay in Releases: The time taken to bring up the entire infrastructure can lead to delays in testing features or conducting sanity checks for bug fixes, resulting in a longer time to market.
High Complexity: Creating and maintaining scripts to standardize environments across different stages can be complex and error-prone
Manual Interventions: Even with automation scripts, manual interventions are often required to configure and install dependencies based on the application's specific requirements, adding to the setup time.
DevOps Dependencies: Developers typically lack expertise in tools like Terraform or Ansible, making them dependent on DevOps or SRE teams to make changes and install dependencies for their applications, which can slow down the development process.
Resource Management: Managing the lifecycle of ephemeral environments can be challenging. These environments need to be deleted once tasks are completed; otherwise, they lead to resource wastage and increased costs.
High Infra Cost: The costs associated with spinning up and maintaining ephemeral environments, particularly in cloud-based setups, can add significantly to the overall infrastructure expenses.

Rethinking Ephemeral Environments

When we talk in terms of Kubernetes, setting up Ephemeral Environments becomes a lot easier than the traditional approach. Kubernetes has a beautiful thing called namespaces, a logical separation of group of resources, providing isolation of workloads within the same cluster.

By leveraging namespaces and some advanced autoscaling methods, it becomes much more easier to create a ephemeral environment that is cost-effective, less complex and helps you dynamically bring up the resources and hibernate when not in use.

How to Set Up an Ephemeral Environment in K8s Manually?

Setting up an ephemeral environment, especially within a Kubernetes ecosystem, involves several key steps that ensure agility, efficiency, and cost-effectiveness. Below, we detail a straightforward approach to creating and managing these temporary environments.

Step 1: Define Your Infrastructure RequirementsBefore you create an ephemeral environment, it's essential to understand the specific requirements of the application or feature being tested. This includes the necessary computing resources, the required services, and any dependencies that need to be replicated from the production environment.

Step 2: Automate the Environment SetupAutomation is crucial in managing ephemeral environments to ensure they can be spun up and torn down efficiently. Tools like Terraform or Ansible can be used to script the creation of your infrastructure. In Kubernetes, you might automate setting up namespaces, deploying container images, and configuring network policies through CI/CD pipelines.

Using Kubernetes NamespacesIn Kubernetes, namespaces provide a way to divide cluster resources between multiple users. Each ephemeral environment can be created in its namespace, isolating its running processes and resources from other environments

Step 3: Deploy Your ApplicationOnce the namespace is ready, deploy your application using Kubernetes manifests or Helm charts. This step often involves setting up the necessary config maps and secrets to configure the application according to the environment

Or using Helm

Step 4: Configure Autoscaling and MonitoringTo optimize costs and resource usage, configure autoscaling for your application workloads. Kubernetes Horizontal Pod Autoscaler (HPA) or a more advanced tool like KEDA can be used to automatically adjust the number of pods based on traffic or other metrics

Monitoring is also essential to track the performance and health of your temporary environment. Tools like Prometheus for monitoring and Grafana for visualization can be integrated to monitor the environment's metrics.

Step 5: Implement Cleanup ProceduresTo ensure that resources are not wasted, set up automatic cleanup procedures to tear down the environment after use. This can be scheduled using cron jobs or integrated into your CI/CD pipeline to destroy the environment once the testing is complete:

Or a more controlled cleanup with Helm

Step 6: Documentation and TrainingFinally, document the entire process and provide training for your teams. This ensures that everyone understands how to efficiently use ephemeral environments, which helps in maximizing the benefits while minimizing potential disruptions or misuse.

Manually creating and deleting the namespaces, and integrating it within pipelines can be something a big pain when it comes to developer productivity. Integrating different tools such as Grafana, Prometheus, Jenkins, ArgoCD, KEDA, etc can be a tedius task for DevOps / SRE engineers as well. With the the involvement of custom scripting, again the complexities increases, high rish to human errors. With Devtron's simplified workflow, it becomes a lot more easier to automate the process, and improve the developer productivity while reducing its high dependencies from DevOps/ SRE teams.

How Devtron Simplifies Ephemeral Environments?

Devtron enhances the management of ephemeral environments through its modern dashboard, simplified workflows, automation and effective cost-management strategies. Here are key features:

Namespace Utilization: In Kubernetes, namespaces provide logical separation, allowing multiple ephemeral environments within the same cluster without additional cost. Devtron leverages this to minimize the overhead associated with setting up and tearing down environments.
Cost Management: Devtron implements strategies such as leveraging spot instances and right-sizing resources, ensuring that the infrastructure costs are kept to a minimum. For example, by using spot instances, organizations can save up to 70-90% compared to standard costs.
Automated Scaling: Devtron employs tools like KEDA for event-driven autoscaling, ensuring resources are used efficiently. Environments can scale down automatically during inactivity and scale up when needed, further optimizing costs.
Simplified Workflow: Devtron provides an intuitive dashboard for all operating on Kubernetes, providing Kubernetes-native CI/CD pipelines, simplifying the heavy scripting and stitching up of different tools to complete an end-to-end workflow.

Along with that, there are many other factors which makes the entire process much more seamless, such as visibility of workloads, application metrics, configurations management, etc.

Setting-up Ephemeral Environments With Devtron

Devtron is a Software Distribution Platform designed for Kubernetes. On its mission to democratize Kubernetes, ephemeral environments are one among the many other features, that make life easier. With Devtron’s intuitive dashboard, operations on Kubernetes become flawless, and it goes with ephemeral environments as well. To get started with ephemeral environment, follow the below steps.

Step 1: Install the keda-add-on-http from the chart’s marketplace. Navigate to the charts store, search for Keda, and as you can see in the below image, you can see all charts related to Keda. Select the appropriate helm chart and deploy it. To add any helm chart which is not listed on the charts store, free feel to check out this blog.

Step 2: Once the controller has been successfully installed, you can see a consolidated view of the deployed helm chart, along with its resources.

Step 3: Now, let’s move and configure the ephemeral environment for my microservice called, payment-svc. To configure the ephemeral environment for any application, the process remains the same and you should be able to configure/ clone the workflows for different applications. Navigate to Workflow Editor, add a workflow for the respective environment where you want to deploy your applications, in our case, its dev-testing environment as you can see in the below image. To understand more about workflows in Devtron, feel free to refer the official documentation.

Step 4: Once the workflow has been created, Devtron automatically creates environment-overrides for the deployment environment. Environment overrides help you manage your Kubernetes configuration for the specific environment in a more efficient way. Under the environment override > dev-testing environment, we can add the relevant configurations required in the deployment template which would create the HttpScaledObject, responsible for bringing the environment up and running dynamically as it receives any HTTP request.

Step 5: After providing the relevant configuration, navigate to Build & Deploy section, select the relevant image, and deploy it in the dev-testing environment. Upon successful deployment, you can see the application status as Healthy, and all details about the deployment are as you can see in the below image.

You can also see all the resources deployed along with the deployment in a resourced grouped view, and perform operations such as checking the logs, events, manifests, or exec into the terminals. You can notice in the below image that, we have a Deployment object but there isn’t any pod running as of now. This is because it automatically scaled down the workload since there is no HTTP request hitting the given hostname/ service.

Step 6: In the Devtron dashboard, it automatically picks up the ingress host and shows it in URLs section at the top right of the dashboard as you can see in Fig. 5, and if any request has been made into the hostname, it will automatically scale up the pod and it can serve the traffic as you can see in the below image.

Conclusion

Ephemeral environments offer a flexible, cost-effective solution for managing development stages, particularly in a dynamic and fast-paced software development landscape. Devtron's approach not only simplifies the management of these environments but also enhances cost efficiency and deployment agility.

Organizations looking to streamline their development processes and reduce costs should consider implementing ephemeral environments, especially those already using Kubernetes. With Devtron, the transition is smoother, allowing teams to focus more on innovation and less on infrastructure management.

Feel free to join our Discord Community if you have any questions. Would love to address any queries or questions. If you liked Devtron, do give it a Star ⭐️ on GitHub.

The Gatekeeper: Why Approval-Based Deployments are Essential for Production Environments on Kubernetes

Devtron — Fri, 13 Sep 2024 09:48:00 +0000

In the fast-paced world of software development, speed and efficiency are paramount. However, when it comes to deploying applications to production environments, particularly on Kubernetes, it’s crucial to prioritize control and security. One effective way to achieve this is through approval-based deployments. In this blog post, we’ll explore the importance of this practice and how it helps maintain the integrity and reliability of production systems.

What is Approval-Based Deployment?

Approval-based deployment is a process where any changes or updates to the production environment must be reviewed and approved by authorized personnel before they are applied. This step acts as a safeguard, ensuring that only vetted and tested code makes its way into the live environment where it can impact end-users. Approval can be made at two different levels, i.e., approval for the configurations and approval for the container image.

The Critical Role of Kubernetes in Modern Deployments

Kubernetes has revolutionized how we deploy, scale, and manage containerized applications. Its flexibility and power make it a favorite among DevOps teams. However, with great power comes great responsibility. Kubernetes allows rapid changes and scaling, but without proper oversight, it can lead to significant risks, including downtime, security vulnerabilities, and performance issues. The adoption of Kubernetes has been increased from 71% to 89% as per CNCF 2023 Annual Survey and out of 89%, 18% are still evaluating Kubernetes to use in production. Though it’s known for its ability to orchestrate a container, Kubernetes does come with a lot of complexities, and using the right set of tools to create an abstraction is very important.

Why Approval-Based Deployments Matter

Approval-based deployments are important for any critical environment. Here are some of the reasons why approval-based deployments matter:

Enhanced Security

Prevent Unauthorized Changes: Approval-based deployments ensure that only authorized individuals can make changes to the production environment. This prevents malicious or accidental modifications that could compromise the production environment.
Review for Vulnerabilities: Each deployment undergoes a thorough review, reducing the risk of introducing security vulnerabilities. This is especially important for Kubernetes environments, where misconfigurations and vulnerable images can lead to severe security breaches.

Improved Stability

Quality Assurance: By requiring approvals, teams can ensure that only stable, tested code reaches production. This minimizes the chances of deploying buggy or unstable applications, which can lead to downtime and user dissatisfaction.
Controlled Rollouts: Approvals allow for controlled, staged rollouts, ensuring that any issues can be quickly identified and addressed without impacting the entire user base.

Compliance and Auditing

Regulatory Compliance: Many industries, primarily Fintech organizations like Banks, are subject to strict regulatory requirements. Approval-based deployments provide an auditable trail of changes, making it easier to demonstrate compliance with these regulations.
Accountability: The approval process holds individuals accountable for the changes they authorize. This promotes a culture of responsibility and thoroughness within the development and operations teams.

Operational Efficiency

Preventing Errors: By catching potential issues before they reach production, teams can avoid costly and time-consuming rollbacks and fixes. This keeps the system running smoothly and efficiently.
Continuous Improvement: The approval process often includes a review and feedback loop, fostering continuous improvement in the deployment process and overall code quality.

Implementing Approval-Based Deployments on Kubernetes

To effectively implement approval-based deployments in a Kubernetes environment, consider the following best practices:

Define Clear Approval Policies: Establish users who can approve deployments with relevant permissions and clear distinction between the user who can approve the container images and critical environment configurations.
Automate the Approval Workflow: Using tools like Devtron to automate the deployment pipeline, integrating approval gates that require manual sign-off at critical environments like production within the workflow.
Monitor and Audit: Implement monitoring and auditing solutions to track changes and approvals. Regularly review these changes to ensure compliance and identify areas for improvement and easy rollback strategies if anything goes wrong with the latest releases.
Continuous Training: Ensure that all team members are trained on the approval process and understand the importance of their role in maintaining production integrity.

Approval-Based Workflows with Devtron

Devtron streamlines the platform engineering workflows for Kubernetes. It provides an intuitive dashboard to deploy and manage all your workloads on Kubernetes through Kubernetes-native CI/CD pipelines as well as Helm Charts, the K8s package manager. With the help of Devtron, users can easily create approval checks for the configurations as well as for the container images for critical environments. Let’s dive into the approval-based workflows with Devtron.

Step 1: Devtron comes with granular RBAC which allows you to define different roles for users, who can approve the configurations and container images before it get deployed in critical environments.

Step 2: After the access has been granted, users can easily create automated workflows with Devtron and integrate the approval checks for configurations and container images. Creating workflows with Devtron is pretty straightforward just with a few clicks and you can create any type of workflows with Devtron, be it sequential or parallel as you can see in the below image.

Additionally adding an approval check is also pretty much straightforward. For the deployment pipeline of any critical environment i.e., prod, simply toggle on the approval checks and define the number of approvals required before you can deploy it. There are no custom scripting or external integrations are required as you can in the below image.

Devtron also comes with Protect Configurations which allows you to protect the configurations of specific environments. For any workflow that you have created, Devtron automatically created Environment Overrides for you as you can see in the [Fig. 3], which ensures the isolation of configurations for multiple deployment environments. To protect the configuration of any environment, simply toggle on the button as you can see in the below image.

If any changes are made in that environment, they first need to be approved by the respective approver, then only those changes can be deployed in the respective environment. Additionally, you can see a diff in the changes made by the user along with the user details. The user asking for approval can also write comments with details on why the changes have been made to provide a proper context to the approver as you can in the below image.

Step 3: After the workflow has been created, and configurations are done, the user can directly move to the Build & Deploy section where the user can trigger the pipelines. If it is an automated pipeline, as soon as you make a new commit, the pipeline will be automatically triggered, which would create a container image, deploy it into the utils environment, perform automated test cases if there are any in the pre/post deployment stages, deploy the same image to the stage environment and before it gets deployed into production, approval is required. To raise an approval request, click on the approval check button as described in [Fig. 3], there you can see all images and additionally, you can also give Image labels and Comment for the change made with this image as you can see in the below image.

The user can select any of the people from the list to approve their container image. The user who is requested will also be notified through email and can approve it via email, additionally, the user can also see the Approval Pending in the dashboard itself with the details of the user who raised an approval request along with labels and comments. We can also see the user who raised an approval request which I can Approve. For some images, I can also Cancel Request, and that’s because I have raised an approval request for those images. The one who raised an approval request cannot approve the self-requests.

It is important to note that, with Devtron you get the entire audit trail of who approved the request, the user who raised a request as well as you can also see the user who has deployed it in the production environment in Deployment History. Additionally, you can also see the details of the configurations of the last deployment, compare them with the older releases and if anything breaks down, can easily rollback with a single click.

Step 4: With Devtron’s single pane of glass, you can also check out the deployment metrics critical for your business such as Deployment Frequency, Change Failure Rate, MTR, and MLT for your production environment as you can see in the below image.

Conclusion

In today’s dynamic software landscape, maintaining control over production deployments is more critical than ever. By implementing approval-based deployments for Kubernetes environments, organizations can enhance security, improve stability, ensure compliance, and boost operational efficiency. This practice not only protects the production environment but also fosters a culture of responsibility and continuous improvement within the team. With Devtron, it becomes much easier to set up guardrails around the Kubernetes ecosystem, natively integrating the gatekeeper within your workflows. As you embark on this journey, remember that the gatekeeper’s role is not to slow down progress but to safeguard the environment, ensuring that only the best, most secure code makes it to production.

If you have any questions, feel free to join our Community Discord Server and shoot out your questions, would be happy to answer them.