Forem: harshaway

Amazon EKS now supports Kubernetes version 1.24 (Eks 1.23 To 1.24 Upgrade)

harshaway — Thu, 13 Jul 2023 05:24:12 +0000

The Amazon Elastic Kubernetes Service (Amazon EKS) team is pleased to announce support for Kubernetes version 1.24 for Amazon EKS and Amazon EKS Distro. We are excited for our customers to experience the power of the “Stargazer” release. Each Kubernetes release is given a name by the release team. The team chose “Stargazer” for this release to honor the work done by hundreds of contributors across the globe: “Every single contributor is a star in our sky, and Amazon EKS extends its sincere thanks to the upstream community and the Kubernetes 1.24 Release Team for bringing this release to the greater cloud-native ecosystem.”

Updating an Amazon EKS cluster Kubernetes version
When a new Kubernetes version is available in Amazon EKS, you can update your Amazon EKS cluster to the latest version.
Update the Kubernetes version for your Amazon EKS cluster
To update the Kubernetes version for your cluster
Compare the Kubernetes version of your cluster control plane to the Kubernetes version of your nodes.

Get the Kubernetes version of your cluster control plane

kubectl version --short

Get the Kubernetes version of your nodes. This command returns all self-managed and managed Amazon EC2 and Fargate nodes. Each Fargate Pod is listed as its own node.

kubectl get nodes

The major changes from eks 1.24 was Amazon EKS ended support for Dockershim
Why we’re moving away from dockershim
Docker was the first container runtime used by Kubernetes. This is one of the reasons why Docker is so familiar to many Kubernetes users and enthusiasts. Docker support was hardcoded into Kubernetes – a component the project refers to as dockershim. As containerization became an industry standard, the Kubernetes project added support for additional runtimes. This culminated in the implementation of the container runtime interface (CRI), letting system components (like the kubelet) talk to container runtimes in a standardized way. As a result, dockershim became an anomaly in the Kubernetes project. Dependencies on Docker and dockershim have crept into various tools and projects in the CNCF ecosystem ecosystem, resulting in fragile code
Docker and Containerd in Kubernetes
Think of the Docker as a big car with all of its parts: the engine, the steering wheel, the pedals, and so on. And if we need the engine, we can easily extract it and move it into another system.

This is exactly what happened when Kubernetes needed such an engine. They basically said, "Hey, we don't need the entire car that is Docker; let's just pull out its container runtime/engine, Containerd, and install that into Kubernetes."

Docker vs. Containerd: What Is The Difference?

Docker was written with human beings in mind. We can imagine it as a sort of translator that tells an entire factory, filled with robots, about what the human wants to build or do. Docker CLI is the actual translator, some of the other pieces in Docker are the robots in the factory.

keep in mind Kubernetes is a program, and Containerd is also a program. And programs can quickly talk to each other, even if the language they speak is complex. containerd is developed from the ground up to let other programs give it instructions. It receives instructions in a specialized language named API calls.

The messages sent in API calls need to follow a certain format so that the receiving program can understand them.

Removal of Dockershim
The most significant change in this release is the removal of the Container Runtime Interface (CRI) for Docker (also known as Dockershim). Starting with version 1.24 of Kubernetes, the Amazon Machine Images (AMIs) provided by Amazon EKS will only support the containerd runtime. The EKS optimized AMIs for version 1.24 no longer support passing the flags enable-docker-bridge, docker-config-json, and container-runtime.

Before upgrading your worker nodes to Kubernetes 1.24, you must remove all references to these flags

The open container initiative (OCI) images generated by docker build tools will continue to run in your Amazon EKS clusters as before. As an end-user of Kubernetes, you will not experience significant changes.

For more information, see Kubernetes is Moving on From Dockershim: Commitments and Next Steps on the Kubernetes Blog.
Kubernetes 1.24 features and removals
Admission controller enabled
CertificateApproval, CertificateSigning, CertificateSubjectRestriction, DefaultIngressClass, DefaultStorageClass, DefaultTolerationSeconds, ExtendedResourceToleration, LimitRanger, MutatingAdmissionWebhook, NamespaceLifecycle, NodeRestriction, PersistentVolumeClaimResize, Priority, PodSecurityPolicy, ResourceQuota, RuntimeClass, ServiceAccount, StorageObjectInUseProtection, TaintNodesByCondition, and ValidatingAdmissionWebhook.

Important changes
Starting with Kubernetes 1.24, new beta APIs are no longer enabled in clusters by default. Existing beta APIs and new versions of existing beta APIs continue to be enabled. Amazon EKS will have exactly the same behaviour. For more information, see KEP-3136: Beta APIs Are Off by Default

In Kubernetes 1.23 and earlier, kubelet serving certificates with unverifiable IP and DNS Subject Alternative Names (SANs) were automatically issued with unverifiable SANs. These unverifiable SANs are omitted from the provisioned certificate. Starting from version 1.24, kubelet serving certificates aren't issued if any SAN can't be verified. This prevents kubectl exec and kubectl logs commands from working. For more information, see Certificate signing considerations for Kubernetes 1.24 and later clusters.

Topology Aware Hints
It is common practice to deploy Kubernetes workloads to nodes running across different availability zones (AZ) for resiliency and fault isolation. While this architecture provides great benefits, in many scenarios it will also result in cross-AZ data transfer charges. You may refer to this post to learn more about common scenarios for data transfer charges on EKS. Amazon EKS customers can now use Topology Aware Hints, which are enabled by default, to keep Kubernetes service traffic within the same availability zone. Topology Aware Hints provide a flexible mechanism to provide hints to components, such as kube-proxy, and use them to influence how the traffic is routed within the cluster.

Pod Security Policy (PSP) was deprecated in Kubernetes version 1.21 and will be removed in version 1.25. PSPs are being replaced by Pod Security Admission (PSA), a built-in admission controller that implements the security controls outlined in the Pod Security Standards (PSS). PSA and PSS have both reached beta feature status as of Kubernetes version 1.23 and are now enabled in EKS. Please read the following when implementing PSP and PSS, please review this blog post.

You can also leverage Policy-as-Code (PaC) solutions such as Kyverno, and OPA/Gatekeeper from the Kubernetes ecosystem as an alternative to PSA. Please visit the Amazon EKS Best Practices Guide for more information on PaC solutions and help deciding between PSA and PaC.

Simplified scaling for EKS Managed Node Groups (MNG)
For Kubernetes 1.24, we have contributed a feature to the upstream Cluster Autoscaler project that simplifies scaling the Amazon EKS managed node group (MNG) to and from zero nodes. Before, you had to tag the underlying EC2 Autoscaling Group (ASG) for the Cluster Autoscaler to recognize the resources, labels, and taints of an MNG that was scaled to zero nodes.

Starting with Kubernetes 1.24, when there are no running nodes in the MNG, the Cluster Autoscaler will call the EKS DescribeNodegroup API to get the information it needs about MNG resources, labels, and taints. When the value of a Cluster Autoscaler tag on the ASG powering an EKS MNG conflicts with the value of the MNG itself, the Cluster Autoscaler will prefer the ASG tag so that customers can override values as necessary.

Change to certificates controller
In Kubernetes 1.23 and earlier, kubelet serving certificates with unverifiable IP and DNS Subject Alternative Names (SANs) were automatically issued with the unverifiable SANs. Beginning with version 1.24, no kubelet-serving certificates will be issued if any SANs cannot be confirmed. This will prevent the kubectl exec and kubectl logs commands from working. Please follow the steps outlined in the EKS user guide to determine if you are impacted by this issue, the recommended workaround, and long-term resolution.

Upgrade your EKS with terraform

Clone below repo from Bitbucket which consists of Terraform files for eks-test cluster

git clone git@bitbucket.org:example.eks.module

Change the EKS version in the terraform.tfvars file In this case, we will change it from 1.22 to 1.23.

$ vi variables.tf

variable "eks_version" {
   default = "1.24"
   description = "kubernetes cluster version provided by AWS EKS"
}

Once all the above modifications are done then execute terraform plan to verify

$ terraform plan
If the changes were made to a single file with launch configuration and auto scaling group (to add would increase by 2 for every file changed), output would be: Plan: 0 to add, 1 to change, 0 to destroy.

1 to change: EKS version from 1.23 to 1.24.

After verification, now it’s time to apply the changes

$ terraform apply
# verify again and type 'yes' when prompted.

Upgrading Managed EKS Add-ons
In this case the change is trivial and works fine, simply update the version of the add-on. In my case, from this release I utilise kube-proxy, coreDNS and ebs-csi-driver.

Terraform resources for add-ons

resource "aws_eks_addon" "kube_proxy" {
  cluster_name      = aws_eks_cluster.cluster[0].name
  addon_name        = "kube-proxy"
  addon_version     = "1.24.7-eksbuild.2"
  resolve_conflicts = "OVERWRITE"
}

resource "aws_eks_addon" "core_dns" {
  cluster_name      = aws_eks_cluster.cluster[0].name
  addon_name        = "coredns"
  addon_version     = "v1.8.7-eksbuild.3"
  resolve_conflicts = "OVERWRITE"
}

resource "aws_eks_addon" "aws_ebs_csi_driver" {
  cluster_name      = aws_eks_cluster.cluster[0].name
  addon_name        = "aws-ebs-csi-driver"
  addon_version     = "v1.13.0-eksbuild.1"
  resolve_conflicts = "OVERWRITE"
}

After upgrading EKS control-plane
Remember to upgrade core deployments and daemon sets that are recommended for EKS 1.24.

Kube-proxy — 1.24.7-minimal-eksbuild.2(note the change to minimal version, it is only stated in the official documentation)
VPC CNI — 1.11.4-eksbuild.1(there is versions 1.12 available but 1.11.4 is the recommended one)
aws-ebs-csi-driver- v1.13.0-eksbuild.1
The above is just a recommendation from AWS. You should look at upgrading all your components to match the 1.24 Kubernetes version. They could include:

cluster-autoscaler or Karpenter
kube-state-metrics
metrics-server

Upgrade AWS Elastic Kubernetes Service (EKS) Cluster Via Terraform 1.22 to 1.23

harshaway — Mon, 22 May 2023 12:26:52 +0000

Kubernetes is the new normal when it comes to host your applications.

AWS Elastic Kubernetes service is a managed service where the control plane is deployed in a High Availability and it is completely managed by AWS in the backend allowing the administrators/SRE/DevOps Engineers to manage the data plane and the microservices running as pods.

As of writing the post today Kubernetes community has a three releases per year cadence for the k8s version. On the other hand AWS has their own customized version of Kubernetes(EKS Version) and have their own release cadence. You could find this information at https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html

Note - EKS upgrade is a step upgrade and can be upgraded from one minor version at a time for e.g. 1.22 to 1.23

Managing AWS EKS via terraform helps us to maintain the desired state and it also allows us seamlessly to perform the cluster upgrade.

Pre-requisites in Terraform
Verify that the state file of EKS does not throws any error before the upgrade.
Ensure the state is stored in a remote place such as Amazon S3
Pre-requisites in EKS
Ensure 5 free IP addresses from the VPC subnets of EKS cluster (explained in below section)
Ensure the Kubelet version is same as the control plane version
Verify EKS addons version and upgrade if necessary before the start of cluster upgrade.
Pod Disruption Budget (PDB) some time cause error while draining pods (recommended to disable it while upgrading)
Use an K8s API depreciation finder tool like Pluto to support the API changes on the newer version.
Upgrade Process
https://docs.aws.amazon.com/eks/latest/userguide/update-cluster.html

Let me break down the upgrade process that happens when we perform the upgrade. This is a sequential upgrade

Control Plane upgrade
The control plane upgrade is an in-place upgrade means that it launches new control plane with the target version within the same subnet of the existing control plane and that is the where we need atleast 5 free IPs in the EKS subnet to accommodate the new control plane. The new control plane will go through readiness and health check and once passed the new control plane will replace the old plane. This process happens in the backend within AWS infrastructure and there will be no impact to application

Node upgrade
The node upgrade is also an in-place upgrade where it launches new nodes with the target version and the pod from old nodes will get evicted and launched in the new node.

Add-ons upgrade
The addons such as coredns, VPC CNI, kube-proxy etc on your cluster need to be upgraded accrodingly as per the matrix in https://docs.aws.amazon.com/eks/latest/userguide/managing-vpc-cni.html#vpc-add-on-update.

Update System Components version (Kube-Proxy, CoreDNS, AWS CNI, Cluster Autoscaler)
Check the system component versions before upgrading. Refer to the below page for the desired version of Kube-Proxy, CoreDNS, and AWS CNI.

Updating an Amazon EKS cluster Kubernetes version - Amazon EKS

Let us take an example of upgrading from 1.22 to 1.23

Step-1:
Ensure control plane and nodes are in same version
kubectl version --short
kubectl get nodes
Step-2:
Before updating your cluster, ensure that the proper Pod security policies are in place. This is to avoid potential security issues
kubectl get psp eks.privileged
Step-3:
Update you target version in your terraform file to the target version say 1.23 and then perform a TF plan and apply

$ vi variables.tf

variable "eks_version" {
   default = "1.23"
   description = "kubernetes cluster version provided by AWS EKS"
}

terraform plan 
terraform apply --auto-approve

Step-4:
Once the control is upgraded,the managed worker nodes upgrade process get invoked automatically. In case you are using the self managed worker nodes upgrade. Choose the AMI as per your control plane version and region in the matrix below https://docs.aws.amazon.com/eks/latest/userguide/retrieve-ami-id.html
Update your worker nodes TF file with the new AMI id and run TF plan and apply

Step-5:
Once control plane and workernodes upgrade were completed. Now it is time to upgrade the addons, see what addons are enabled in your cluster and upgrade each addons via console or eksctl based on how you manage it.
Each addons has the compatiblity matrix from the AWS documentation and it has to be upgraded appropriately
sample ref : https://docs.aws.amazon.com/eks/latest/userguide/managing-vpc-cni.html#vpc-add-on-update

Step-6:
it's mandatory to install AWS-EBS_CSI Driver for eks cluster from 1.23 to next versions Follow the below steps to upgrade the EKS cluster to version 1.23.

Need to Deploy Amazon EBS CSI Driver follow below steps.

Creating the Amazon EBS CSI driver IAM role for service accounts - Amazon EKS

Using AWS Management Console:

To create your Amazon EBS CSI plugin IAM role with the AWS Management Console
Open the IAM console at https://console.aws.amazon.com/iam/.

In the left navigation pane, choose Roles.

On the Roles page, choose Create role.

On the Select trusted entity page, do the following:

In the Trusted entity type section, choose Web identity.

For Identity provider, choose the OpenID Connect provider URL for your cluster (as shown under Overview in Amazon EKS).

For Audience, choose sts.amazonaws.com.

Choose Next.

On the Add permissions page, do the following:

In the Filter policies box, enter AmazonEBSCSIDriverPolicy.

Select the check box to the left of the AmazonEBSCSIDriverPolicy returned in the search.

Choose Next.

On the Name, review, and create page, do the following:

For Role name, enter a unique name for your role, such as AmazonEKS_EBS_CSI_DriverRole.

Under Add tags (Optional), add metadata to the role by attaching tags as key–value pairs. For more information about using tags in IAM, see Tagging IAM Entities in the IAM User Guide.

Choose Create role.

After the role is created, choose the role in the console to open it for editing.

Choose the Trust relationships tab, and then choose Edit trust policy.

Find the line that looks similar to the following line:

"oidc.eks.region-code.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:aud": "sts.amazonaws.com"
"oidc.eks.region-code.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:sub": "system:serviceaccount:kube-system:ebs-csi-controller-sa"

if we are using encryption on ebs then we need to sepearte policy and role for the same(attach the steps needed )like below

Copy and paste the following code into the editor, replacing custom-key-arn with the custom KMS key ARN.


{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "kms:CreateGrant",
        "kms:ListGrants",
        "kms:RevokeGrant"
      ],
      "Resource": ["custom-key-arn"],
      "Condition": {
        "Bool": {
          "kms:GrantIsForAWSResource": "true"
        }
      }
    },
    {
      "Effect": "Allow",
      "Action": [
        "kms:Encrypt",
        "kms:Decrypt",
        "kms:ReEncrypt*",
        "kms:GenerateDataKey*",
        "kms:DescribeKey"
      ],
      "Resource": ["custom-key-arn"]
    }
  ]
}

Choose Next: Tags.

On the Add tags (Optional) page, choose Next: Review.

For Name, enter a unique name for your policy (for example, KMS_Key_For_Encryption_On_EBS_Policy).

Choose Create policy.

In the left navigation pane, choose Roles.

Choose the AmazonEKS_EBS_CSI_DriverRole in the console to open it for editing.

From the Add permissions drop-down list, choose Attach policies.

In the Filter policies box, enter KMS_Key_For_Encryption_On_EBS_Policy.

Select the check box to the left of the KMS_Key_For_Encryption_On_EBS_Policy that was returned in the search.

Choose Attach policies.

Managing the Amazon EBS CSI driver as an Amazon EKS add-on
An existing cluster. To see the required platform version, run the following command.

aws eks describe-addon-versions --addon-name aws-ebs-csi-driver
To add the Amazon EBS CSI add-on using eksctl

eksctl create addon --name aws-ebs-csi-driver --cluster my-cluster --service-account-role-arn arn:aws:iam::111122223333:role/AmazonEKS_EBS_CSI_DriverRole --force

Updating the Amazon EBS CSI driver as an Amazon EKS add-on
Amazon EKS doesn't automatically update Amazon EBS CSI for your cluster when new versions are released or after you update your cluster to a new Kubernetes minor version. To update Amazon EBS CSI on an existing cluster, you must initiate the update and then Amazon EKS updates the add-on for you.

To update the Amazon EBS CSI add-on using eksctl
Check the current version of your Amazon EBS CSI add-on. Replace my-cluster with your cluster name.

eksctl get addon --name aws-ebs-csi-driver --cluster my-cluster

The example output is as follows.

NAME VERSION STATUS ISSUES IAMROLE UPDATE AVAILABLE

aws-ebs-csi-driver      v1.11.2-eksbuild.1      ACTIVE  0               v1.11.4-eksbuild.1

Update the add-on to the version returned under UPDATE AVAILABLE in the output of the previous step.

eksctl update addon --name aws-ebs-csi-driver --version v1.11.4-eksbuild.1 --cluster my-cluster --force

by the above procedure, ebs-csi-driver installation will be completed.