Forem: Jonathan Kaftzan

How to Deploy the Kubernetes Dashboard

Jonathan Kaftzan — Thu, 17 Nov 2022 13:53:33 +0000

Kubernetes clusters consist of multiple resources and API objects interacting dynamically—which typically makes cluster management via the CLI overwhelming. Kubernetes Dashboard was built to simplify cluster operations by providing a unified, human-friendly interface. The web-based dashboard enables cluster operators to deploy applications, access running workloads, and correlate logs with cluster events. This article demonstrates how to install and use Kubernetes Dashboard.

What Is the Kubernetes Dashboard?

Kubernetes Dashboard offers a web-based UI to manage your Kubernetes resources and applications. It displays various macro details of a cluster including nodes, namespaces, volumes, job details, roles, and deployments.

The dashboard presents an intuitive interface to interact with the cluster’s API server, enabling various functions, which we discuss below.

Deploying Applications

Kubernetes Dashboard provides a wizard to deploy containerized workloads within a deployment. The wizard allows for manual specification of the deployment’s details; you can also automate this by uploading manifest files.

Managing Cluster Resources

The admin overview displays all namespaces, persistent volumes, and nodes running in the cluster. Each resource of the cluster is displayed alongside associated metrics, specifications, and events.

The dashboard includes separate views for: workloads, services, storage, ConfigMaps, secrets, and logs, allowing cluster operations teams to drill down into specific information for container resource management.

Troubleshooting Kubernetes Workloads

The dashboard offers insights into the cluster state, any errors that may have occurred, and their related events. It also provides a log viewer linked to detail pages and pod lists. You can use this to measure performance benchmarks and workload bottlenecks at the container level.

Although Kubernetes Dashboard stands out as a resourceful platform for beginner practitioners running test clusters, it is considered inefficient for complex production clusters since it lacks extensible authentication support. In large clusters with multiple deployments and namespaces, the dashboard performs sub-optimally and requires a larger memory allocation.

Kubernetes Dashboard Alternatives and Integrations

As previously mentioned, Kubernetes’ integrated dashboard does not scale well to provide security guarantees for large production clusters. The tools below can be used to extend the dashboard or provide dashboard-like capabilities for enterprise workloads.

Prometheus

Prometheus is one of the most popular open-source monitoring platforms that collects metrics from various sources (e.g., databases, API servers, etc.), stores them in a time-series database for historical representation, and exposes them via HTTP endpoints. Although the platform is designed to be easy to use, it is powerful enough to provide deep insights into how your applications are performing.

Prometheus is often used with Grafana for comprehensive monitoring of distributed systems. While Prometheus collects, aggregates, and stores metrics about the state of your system, Grafana is an open-source web interface for visualizing cluster health and performance bottlenecks in real time.

Commonly considered the default alternative to Kubernetes Dashboard, the Prometheus-Grafana stack can be installed in a few simple steps through either Helm charts or the Prometheus Operator. The Operator further simplifies the deployment while offering different configuration options for the observability of varying cluster complexities.

ContainIQ

The ContainIQ monitoring and tracing solution monitors cluster events, metrics latency, and logs. It ships with pre-built dashboards to expedite cluster health monitoring, while the Kubernetes events monitoring dashboard keeps a historical record of all cluster events and provides alerts for common events like pod eviction and CrashLoopBackOff.

ContainIQ also allows cluster operators to search and filter logs by timestamp, message, cluster, or pod depending on the level of granularity required.

Portainer

Portainer’s Kubernetes management platform provides a central environment to configure, manage, and secure Kubernetes clusters. The intuitive, open-source tool includes native Kubernetes RBAC support with five predefined roles that administrators can assign to users. Portainer also ships with a quota feature that simplifies the assignment of CPU, RAM, and network capacity limits for fair resource allocation.

Key features of Portainer that make it a suitable alternative to Kubernetes Dashboard include:

GitOps automation

Cluster segmentation
Option to allow/disable access to a default namespace
Permissions for administrators to enable/disable resource over-commitment
Lens IDE
Lens is an integrated development environment that provides complete situational awareness for resources and workloads running in Kubernetes. Lens Kubernetes offers managed dev clusters that offload most of the operations required to keep workloads running. Lens desktop provides a way for offloading local clusters to the cloud without any difference in user experience.

The service also includes an intelligent auto-shutdown feature that reduces cost of ownership by terminating idle resources. Operators can additionally deploy self-managed clusters using Lens, enabling flexibility while guaranteeing high security and availability.

Kubescape

Kubescape is an open-source Kubernetes platform that offers a CLI or a web-based interface for securing, monitoring and managing Kubernetes environments like clusters, manifest files (e.g. YAML, Helm), code repositories, container image registries etc. Along with enabling comprehensive monitoring of cluster health, the platform also helps extract detailed information about each cluster node at the granular level.

Kubescape offers a single-pane-of-glass view of K8s components, including pods, services, deployments, ReplicaSets, ingresses, etcd servers, and much more. Through a web interface to the cluster API, Kubescape also allows you to perform common operations such as creating/deleting nodes and scaling up or down. While Kubescape offers rich features in monitoring and managing cluster components, the platform is primarily used for vulnerability assessment, compliance, identifying configuration drifts, and automated security scanning.

K9s

K9s is a powerful terminal UI that makes it easy to navigate, observe, and manage containerized Kubernetes workloads. The platform allows developers and administrators to define command shortcuts via hotkeys and aliases to expedite cluster management processes. Via pulses and XRay view, the UI offers granular visualization of resources when orchestrating a cluster.

K9s ships with built-in performance benchmarking to ensure pods and services are assigned with optimum resources and limits.

Steps to Deploy Kubernetes Dashboard

Although this demo uses a lightweight minikube setup for the installation and use of Kubernetes Dashboard for cluster resource management, you can follow the same steps for a production-grade Kubernetes cluster. As a prerequisite, the demo requires access to the kubectl CLI to deploy the dashboard’s manifest and enable authenticated access.

Deploying the Dashboard Using a Manifest

Connect with the Kubernetes cluster using one of the CLI client terminals, and deploy the recommended configuration for the dashboard by running the command:

$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.5.0/aio/deploy/recommended.yaml

Upon successful application, the terminal will display the following output:

namespace/kubernetes-dashboard created serviceaccount/kubernetes-dashboard created service/kubernetes-dashboard created secret/kubernetes-dashboard-certs created secret/kubernetes-dashboard-csrf created secret/kubernetes-dashboard-key-holder created configmap/kubernetes-dashboard-settings created role.rbac.authorization.k8s.io/kubernetes-dashboard created clusterrole.rbac.authorization.k8s.io/kubernetes-dashboard created rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created deployment.apps/kubernetes-dashboard created service/dashboard-metrics-scraper created deployment.apps/dashboard-metrics-scraper created

Confirm the deployment of dashboard resources by listing the pods running within the cluster using the command:

$ kubectl get pods -A
Which returns a response similar to:

NAMESPACE NAME READY STATUS RESTARTS AGE ingress-nginx ingress-nginx-admission-create--1-mzfh6 0/1 Completed 0 17m ingress-nginx ingress-nginx-admission-patch--1-2ccwx 0/1 Completed 0 17m ingress-nginx ingress-nginx-controller-5f66978484-8rmlp 1/1 Running 0 17m kube-system coredns-78fcd69978-996nt 1/1 Running 0 20m kube-system etcd-minikube 1/1 Running 0 21m kube-system kube-apiserver-minikube 1/1 Running 0 21m kube-system kube-controller-manager-minikube 1/1 Running 1 (21m ago) 21m kube-system kube-proxy-2ffz6 1/1 Running 0 20m kube-system kube-scheduler-minikube 1/1 Running 0 20m kube-system storage-provisioner 1/1 Running 0 20m kubernetes-dashboard dashboard-metrics-scraper-c45b7869d-rp4gl 1/1 Running 0 12m

Authenticating to Kubernetes Dashboard

The default setting for Kubernetes Dashboard is only minimally configured for RBAC. To log into the dashboard, Kubernetes supports the use of a bearer token, which you can generate by first creating a user service account with administrative privileges. Create the user by specifying the ServiceAccount and ClusterRoleBinding in a file named dashboard-user.yaml with specifications similar to:

apiVersion: v1 kind: ServiceAccount metadata: name: admin-user namespace: kubernetes-dashboard
—

`
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: admin-user
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
– kind: ServiceAccount
name: admin-user
namespace: kubernetes-dashboard

Apply the above configuration to the cluster:

$ kubectl apply -f dashboard-user.yaml

`
`
Which returns a response similar to:

`
serviceaccount/admin-user created
clusterrolebinding.rbac.authorization.k8s.io/admin-user created

Then, generate the bearer token by running the command:

$ kubectl -n kubernetes-dashboard get secret $(kubectl -n kubernetes-dashboard get sa/admin-user -o jsonpath=”{.secrets[0].name}”) -o go-template=”{{.data.token | base64decode}}”
`
`

Which outputs a token similar to the code in the image below (NB: Parts of the output have been omitted for brevity):

Copy this token and store it safely, as it will be used to access the web UI.

Accessing Your Kubernetes Dashboard

First, expose the cluster dashboard to your local machine by running the command:

$ kubectl proxy

Which returns a response of the form:

Starting to serve on 127.0.0.1:8001

The above output implies that the dashboard is accessible through the URL:

http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/

Copy and paste this link into the browser to access the dashboard login page. The page will prompt you for the access token:

Paste the bearer token into the Enter token section and then click Sign in.

Using Kubernetes Dashboard

A successful login will redirect you to a welcome page of the Kubernetes cluster. Note that initially there will be no resources to display:

Deploying a Containerized Application

To deploy a new containerized application, click on the + sign toward the top-right area of the dashboard. This offers three options:

Create from input uses a YAML file pasted directly into the console.
Create from file uses a YAML/JSON file uploaded to the dashboard
Create from form allows for the creation of deployment objects by entering values into a form. This is an intuitive UI workflow, and the one we will use for this demo.

Click on Create from form to open the interactive deployment UI:

Create a simple app running an Nginx image with three replicas by inputting the values shown below:

Once the details are confirmed, click on Deploy to deploy the workload into the cluster.

Navigating the Dashboard

After deploying the application, the dashboard populates the Workloads view with the list of pods, replicas, and deployments, as shown:

Navigating to specific workloads involves a simple, click-based workflow. For instance, to access deployments, select the Deployments option from the Workloads menu:

To access the pods, select the Pods menu from the Workloads menu:

You can also use the dashboard to access, observe, and manage Services, Application Details, Cluster Events, and several other key metrics for observing the performance of a Kubernetes cluster:

Summary
Kubernetes Dashboard provides an intuitive, web-based UI that simplifies the deployment and management of your Kubernetes applications. It enables quick visibility and a click-based workflow for easier cluster management by users of all proficiency levels.

While the default dashboard offers basic features for smaller, non-production cluster management, observability tools such as Lens, ContainIQ, K9s, and Portainer extend the functionality of Kubernetes Dashboard, simplifying the monitoring, deployment, and management of production-grade, complex clusters at a global scale.

How to validate Kubernetes YAML files?

Jonathan Kaftzan — Thu, 20 Oct 2022 10:15:35 +0000

Kubernetes has taken center stage in how we now manage our containerized applications. As a result, many conventions to define our Kubernetes apps exist, including structures such as YAML, JSON, INI, and more.

This leaves us to consider what is the best strategy to follow for our applications. Additionally, we must then also ask how we can validate our application configurations depending on the path we’ve chosen in terms of file structure and especially security.

In this piece, we will explore defining Kubernetes applications using YAML configs and the various steps we can take to effectively validate these config definitions.

Kubernetes Config Definitions in YAML

YAML, compared to JSON and INI, is much more compact and readable. For example, if we were to define a pod that can be reached on port 80, then the configuration in YAML, JSON, and INI would be as shown in the table below.

It is clearly visible that YAML simplifies how we can define our Kubernetes applications, especially considering that a normal application can involve dozens of config files. Additionally, the compact nature of YAML allows you to group objects together, reducing the number of files required.

However, there are major challenges with defining our Kubernetes config files, particularly when attempting to embed constraints and relationships between manifest files. For example, how can we ensure that the memory limits are configured to follow best practices?

Not only can the lack of validation result in unexpected behavior from our applications when edge cases are met, but it can also expose major security loopholes. Hence, it is necessary that we consider validation strategies for our YAML-based config files, and that’s what we will dive into in the following sections.

Various Aspects of Validation

There are three levels of validation that should be performed on our YAML files. These levels ensure that validation is performed in terms of the actual validity of the YAML file all the way to whether or not security practices have been met.

The first level is structural validation, which is the highest level of validation performed on Kubernetes configuration files. It involves simply validating the YAML file to ensure there are no syntax errors when writing it. This is something that can be validated by IDEs used when writing the config files.

The second level is semantic validation. This ensures that the content of the YAML file translates to the Kubernetes resources desired, hence validating the Kubernetes application itself.

Finally, the third and deepest level of validation is security validation, to ensure that the Kubernetes application defined does not have any vulnerabilities. We may have been successful in writing our YAML config successfully to achieve the required Kubernetes resources and connections, but this does not ensure our Kubernetes application is well-secured and follows best practices. The last two validations above are both Kubernetes configuration validations, and not only in the aspect of YAML format validation. Since they are application-specific validations, a dedicated

treatment is necessary. Deep and professional knowledge of the Kubernetes domain is required to perform such validations, and we will take a short look at how to handle them easily using tools developed by Kubernetes domain experts.

For example, locking down hostPath mount permissions makes sure a container in the cluster with a writable hostPath volume is not accessed by attackers, as they may then gain persistence on the underlying host. This does not follow security best practices, and to avoid the issue, we should always ensure that the readOnly section under the hostPath attribute is set to true.

Another example is granting hostNetwork access to pods only when it is necessary. All pods that have access should also be whitelisted. Unnecessary access to hostNetwork increases the potential attack surface area.

So, it can be seen from the two examples above that even after our config files have passed the structural and semantic validations, leading to our Kubernetes resources successfully being orchestrated, security and functional vulnerabilities may still exist. Hence, we must consider how best to capture these vulnerabilities before being alerted to the consequences in production. Security validation is the method to do this.

Validation and Best Practices

Considering that structural validation is fairly simple, it can be expected that the tooling available is now readily available in your IDEs. Kubernetes semantics and security validations require a bit more effort, especially in terms of strategy and tooling.

Let’s consider some of the best practices and strategies these entail to achieve overall validation of our YAML files.

Always Shift Left

We may always validate our K8s configurations, even though we already deployed them in the cluster, using tools like Kubescape, but it’s always better to “shift left”. Shift left is a term that has blown up in the past five or six years, and rightly so. You want to bring validation closer to the delivery and build stage of your applications.

For example, semantic validation is something that Kubernetes itself can handle. However, by the time you leave it to Kubernetes to handle, it’s too late.

You could perform a dry run (kubectl apply -f --dry-run='server’') to validate the semantic structure, but this still is an additional step that could slow down your overall velocity. However, a dry run requires you to have access to the Kubernetes cluster.

An alternative to this approach is Kubeval, which is an amazing tool that can be used to validate your config file semantics to ensure they meet Kubernetes’ object definition requirements. It can be part of your CI process and perform scanning locally, thereby ensuring that you semantically validate your config files before you go to production.

Security validation can also be achieved in the CI using Kubescape. This is an open-source tool that ensures your Kubernetes application definitions follow multiple security frameworks such as NSA-CISA or MITRE ATT&CK®. By using the Kubescape CLI, you can scan all of your YAML files for security vulnerabilities and even get a risk score and risk trends. It acts as a YAML validator with its main value being security validation.

DevOps to DevSecOps

You can run a combination of tools already discussed in your CI pipeline to achieve structural, semantic, and security validation. However, just utilizing these tools and their predefined checks is not enough.

One thing we’ve learned from DevOps is that there is always a cycle of improvement that can be adopted. That is why as your application evolves, and your security needs change, you should constantly review your security controls.

New security controls are something you should incorporate into your security validation steps. Kubescape, an open-source platform by AMRO, allows you to define your own framework of controls. Even though the out-of-the-box framework is robust, you will see your policy controls take shape per the exact needs of your business and Kubernetes resources.

Such a practice can only be achieved if security concerns are embedded into the culture of how you’re building apps. Thanks to open-source tools such as Kubeval and Kubescape, the barriers to your development team continuously thinking about validation, and especially security validation, has been lowered.

Conclusion

YAML config files have made building Kubernetes applications quite simple. However, YAML does have its limitations in terms of validation. It is therefore necessary that we are aware of all validation strategies to ensure that we build our Kubernetes applications to be healthy and secure.

Validating the structure of a YAML file is fairly simple with YAML tests in any IDE, but validating the correctness of Kubernetes resource object definitions and the security measures around them is difficult. Luckily, tools such as Kubescape close the gap, allowing you to shit left and constantly think about security across your entire application lifecycle.

With security being one of the prime concerns of building containerized applications today, the validation strategies discussed here are one step in the right direction.

Securing CI/CD Pipelines Through Security Gates with Kubescape

Jonathan Kaftzan — Sun, 02 Oct 2022 07:09:46 +0000

DevOps and modern engineering have enabled us to provide higher quality code at greater speeds by introducing guardrails and checks into our automated continuous integration (CI) and continuous deployment (CD) processes. However, with security becoming a more pressing matter as more critical zero-day threats arise and misconfigurations are introduced to production systems, while at the same time as application and development processes all moving to more automated CI/CD processes––this is becoming a critical point for enforcing security validations and checks.

With DevSecOps coming a long way as a discipline, there are now great frameworks and best practices for applying security gates in your CI, and later CD.

We’d like to provide a practical guide for doing so in cloud-native environments, and particularly Kubernetes environments. These security gates will enable you to detect early and prevent known security issues from reaching production, such as documented vulnerabilities, common misconfigurations, permissions, and others. CI/CD Security gates include not only the technical controls that need to be applied but also the way we think about security posture, risk appetite, and how to continuously enforce cloud-native security in an ongoing manner.

We believe that security can be embedded as early as your first line of YAML, upon pulling your container images, scanning, deploying, and then even in ongoing post-deployment continuous security monitoring; all with open source tooling. In addition, with the ability to leverage widely adopted developer tooling from VSCode, CLIs, and GitHub Actions, all of these checks can be included from within existing developer workflows and context.

CI Security Hygiene Needs a Hard Reset

CI security receives far less attention than other areas of application and cloud security. Let’s not even talk about security, let’s take quality as an example. You have an endless number of linters or compilers that will return warnings if there are errors in the code. There are even grammar-checking tools for README files (A LOT!), probably even more than those that check for security. In the same way, we as developers, are obsessed with our code and grammar hygiene, how are we not equally concerned with our CI security hygiene?

As we all know and have heard many times through the “shift left” manifesto, the earlier we apply security in the process the better this will be from every kind of perspective––cost, effort, and time. However, likely the most critical piece is catching issues early before they propagate and manifest to production, but also with sufficient context (for non-security experts) to actually be able to do something meaningful about it.

There is also a common misconception that security needs to be a heavy lift, and bog down development processes, and I’d like to bust some of these myths and enable you to adopt security as you write your code, through your CI and CD, rinse and repeat.

_Common CICD Pipeline
_

Security Gate #1: Prevention During Coding

Security hygiene starts as early as the first lines of code you write – whether it’s JSON, YAML, Helm charts, or anything else, there are well-known misconfigurations and vulnerabilities that can be caught during coding.

Kubescape VSCode Extension

With the open-source Kubescape VSCode extension, you receive in-editor notifications while writing YAML about potential security issues by marking the relevant lines in your manifest files. This saves the extra step of having to scan your config files after you have already completed the work and enables developers while still in context to make immediate changes and edits.

In addition, these alerts can also enforce security hygiene based on known and popular security frameworks like NSA-CISA and MITRE ATT&CK and other K8s-related compliance frameworks. You can also build an internal or custom security framework, that contains common misconfigurations that can be custom-defined and continuously updated, such as avoiding privilege escalation through least-privilege practices, and best practice resource limitations.

_Highlighting potential security issues while doing in VS Code with Kubescape
_
By proactively adding security controls into your coding, you can minimize the number of vulnerabilities your scanning tools will later find, and lower the noise level. This matters in the context of security scanning tools, as they will always find misconfigurations and vulnerabilities––and this creates a lot of alert fatigue and what we call “CVE Shock”. This overwhelms developers – not knowing where to get started with remediating these known vulnerabilities and issues.

Security Gate #2: Detection Through Code Repository Scanning

After we’ve written the configuration code, the natural next step is to push it through our CI (usually via the CLI) and try to get our Pull Request (PR) merged into our codebase. However, a good security practice is to nonetheless scan your public and private code repositories and your container image registries before deploying to production, as there are always additional security considerations with your stack, tools, supply chain, and registries.

Kubescape CI Config & CLI

This is where the Kubescape CLI comes in handy.

The Kubescape CLI can scan any of the common Kubernetes / CI configuration files including YAML, JSON, and even their output in the XML & JUnit formats. It can also scan your config based on common and well-known security frameworks like MITRE, NSA, or CIS, or by your own custom-defined security framework of choice.

Due to the fact that the threat and vulnerability landscapes are constantly growing, be forewarned, it will always find vulnerabilities (sometimes many), that are then categorized by severity. This is when good security practices and processes are important.

We’ll start by saying that you DO NOT need to resolve and remediate every issue immediately, but a security best practice is to acknowledge all of the vulnerabilities and to create exceptions when you choose not to fix them. This way there is a log for the process, as well as a decision owner, and full context to the vulnerability.

This means that whenever you run a security scan, you should always do the following once there are findings:

Acknowledge the findings (in a log)
Fix or Ignore them
In addition to this, you can also define many different parameters that enable you to make the right fix / ignore security decisions, such as:
Risk threshold
When to fail/pass a build (based on severity or the risk threshold defined)
Custom frameworks
Exceptions
This is the GitOps way to apply security from a CI/CD perspective and enables the ongoing prioritization of security threat remediation based on how lucrative the fix is.

Scanning Kubernetes workload definitions files (YAML) with Kubescape CLI

Kubescape & GitHub Actions

GitHub Actions are becoming a popular way to configure simple pipelines and CI processes for GitHub repositories. Therefore, in the same way, that you can leverage Kubescape, you can also utilize GitHub Actions.

On top of the same features and capabilities you receive from the CLI, the GitHub Actions support also makes it possible to receive visualizations and graphical representations right inside your GitHub repository. It also enables the CI security scanning to happen directly inside developer workflows and context, and not need to add any additional solutions.

_Example: Running Kubescape from GitHub actions
_

_Example: Showing Kubescape scan results in GitHub
_

Security Gate #3: Detection Through Container Image Registry Scanning

One commonly accepted security best practice is ensuring that any resources we take off the public web are thoroughly scanned, to ensure the end-to-end security of everything in our software supply chain. And this isn’t limited to just our code imports and packages, but also to the containers we use and the registries we employ.

Another thing we have learned along the way is that if security isn’t embedded into existing processes, it very often simply won’t happen. That is why you need to have container image registry scanning that enables you to scan images directly from their registries (e.g. ECR, GCR, quay.io and more) before they are deployed and run in the cluster.

This is an incredibly important security guardrail that can be easily applied and automated, to ensure that vulnerabilities are detected during the development process from third-party registries, and like the previous guardrails––have some mechanism in place to prevent vulnerabilities from reaching deployments and production environments from this vector, as well.

_Container image registry scanning by Kubescape cloud
_
This leads us to our next guardrail, as security is never simply a one-off scan.

Security Gate 4: Continuous Security Post-Deployment

We’ve spoken about good practices and tooling when it comes to CI security, but security is a living, breathing, and continuous process, and it doesn’t end once your PR is merged. We need to have the proper guardrails in place, to monitor for ongoing threats as they emerge.

Therefore, while we scanned our registries during our CI security process, what happens if threats emerge for container registries and configurations already running in production? This is where CD security good practices are required.

By proactively requiring security scanning upon a compelling event (e.g. if someone changed something in the system), according to a predefined schedule (daily, weekly, monthly), or when a security vulnerability is discovered––you can detect if the CVE exists in your production systems or discover potential configuration drift. This means you don’t have to have already strapped security engineers continuously monitoring for security threats, you can create a greater sense of ownership for security on the entire team, through continuous and ongoing security that will alert you to new threats in your running operations, without any specific security domain expertise.

By ensuring CI security starts as early as the code, then adding gates before deployment and more checks post-deployment you can provide end-to-end security controls throughout your CI/CD processes, and minimize the potential of vulnerabilities leaking into production systems. When we apply security controls early, and natively into development processes, we create shared ownership and understanding that security cannot be an afterthought with the complexity of cloud-native systems.

Embedding security gates into CI/CD pipeline

CI/CD Security: How to Keep your CI/CD Pipelines Secure

Jonathan Kaftzan — Wed, 28 Sep 2022 14:50:59 +0000

The tech sector is expanding aggressively. And for businesses and services, keeping up with this growth implies only one thing: integrate new technologies and innovate at pace or be left out.

The growth in the past in this sector has resulted in numerous open-source programming languages and frameworks that are easily obtainable by enterprises to benefit from and meet customer expectations. As simple as this may sound, launching stable and secure software or products to competitive markets is only what happens at the surface. On the surface, bringing continuous improvements, scalability, and security; testing features; configuring specifications, and manually integrating services over time introduces unique challenges to software development.

DevOps principles offer an ingenious solution to tackle these challenges. DevOps is a set of practices that emphasize the collaboration and communication of both software developers and other information technology (IT) professionals while automating the process of software delivery and infrastructure changes. It comes packed with a perfect blend of tool sets, practices, and cultural philosophies for development and operations with an added advantage of increased efficiency, delivery speed, and overall security.

CI/CD sits at the core of DevOps. The main aim of CI/CD is to automate and streamline the software development process by making small changes and additions incrementally. It helps in pushing out features faster with fewer errors. When implemented correctly, it generalizes the complete develop and deliver cycle with the you-build-it-you-own-it pipeline concept, and rapidly makes features market ready. It amplifies developer productivity and extensively helps fulfill organizational goals.

This post will cover everything about CI/CD and CI/CD security, as well as some common threats and best practices.

What is CI/CD?

As mentioned earlier, CI/CD is an integral part of DevOps. DevOps teams use it to push out features and updates quickly, frequently, and with fewer errors. In simple words, CI/CD is a method of automating the software development process so that small changes can be made incrementally.

CI/CD (continuous integration/continuous delivery or deployment) is the foundation on which DevOps processes operate. CI/CD enhances the pace of continuously packaging client and business requirements to deliver an end product with a whiff of extensibility and performance boost. Although widely referred to as the same, DevOps and CI/CD are two completely different methodologies.

DevOps enables development teams to streamline and automate processes/workloads with proven approaches and is at the heart of the software development life cycle. CI/CD allows teams to constantly create new features and push features onto a central repository to deploy faster to meet deadlines.

CI/CD focuses on implementing workflows and automating everything. The general rule of thumb is that manual tasks with repeated iterations need to be automated. Bringing together production readiness and automation with the use of meaningful and secure integrations with third-party software and libraries in no time is where CI/CD shines. In CI/CD, automation is favored over manual processes to achieve speed, agility, and reliability.

Without CI/CD, the team silos tend to make the process tedious and slow. CI/CD effectively aims to break down silos between developers and operations to make the plan depicted above continuous and without friction.

The other integral part of CI/CD and DevOps is collaboration. To achieve successful collaboration, it is important to have a shared vision and agreed-upon objectives. Everyone should be aware of what needs to be accomplished and why it is essential. Furthermore, it is critical to break down silos and have a common understanding of the development process.

What is a CI/CD Pipeline?

The CI/CD pipeline represents an agile set of streamlined tasks that are chained together to generate workflows, some cyclic and some acyclic.

The whole intent of a CI/CD pipeline is to facilitate the execution of end-to-end stages of application delivery in an efficient and errorless routine. These pipelines interlace and automate all phases of the software development life cycle so that all stages from development to deployment are seamlessly triggered through a single execution point.

A CI/CD pipeline can be as simple as developing an evolving workflow to move a file from a source to a destination with new integrations and security patches or as complex as building an entire software service iteratively.

Benefits of a CI/CD Pipeline

When executed diligently, CI/CD delivers numerous benefits to the business and end users, making it one of the most sought-after practices in modern application delivery. The following are a few of the many benefits CI/CD pipelines offer.

Minimal time to develop and deploy

The prime goal of CI/CD implementation is to quicken software development with iterative updates. It promises to deliver features to users at a much faster rate with lesser time spent on the development process.

With a managed and automated CI/CD pipeline, developing and deploying changes to production becomes a regular and non-eventful task that can be accomplished in a matter of hours or days. The development process is no longer stalled waiting for the next big release as features can be deployed to production as soon as they are ready.

Improved code standards and quality

Along the integration process, development teams combine and implement the best practices and standards of the modern and internal proven libraries into their tech stack, improvise as they go along, document and maintain better performing standards for upcoming projects and extend the performance and quality.

With automation being the centerpiece of CI/CD, developers can automate and reuse tests across different workflows and diverse scenarios. This benefit of CI/CD pipelines enables code reusability by handling repeated builds, tests, and environmental deployment tasks.

3 C’s Maximized (Collaboration, Creativity, and Communication)

CI/CD enforces coordination across cross-functional teams by breaking operational silos. A team’s flexibility in collaborating efficiently enhances a feedback loop that offers performant and efficient alternatives to known solutions.

Periodic collaboration between multiple stakeholders also eliminates requirements bottlenecks and allows teams to develop independently by avoiding conflicts. Creativity and communication are further amplified by the ability to develop and test code rapidly without compromising quality.

2 R’s Minimized (Risk and Review Time)

Maintaining code in a central version control system opens the door to extending the code quality with tests and formatting. This ensures that on every pull request, the code gets validated against tests, pull requests undergo review by the admin, and changes are merged strictly upon pass status. The approach also helps identify code bugs before the merge and gives a better overview of what to refactor and where.

The reduced risk and review make CI/CD a promising approach to minimizing security and configurational flaws, enabling developers to launch a safe and reliable product to the market.

Easy production deployment and rollbacks

CI/CD is best at carrying out live incremental changes. Deploying code changes iteratively highlights the most common pain points and mistakes the development teams undergo. Developers can overcome these problems by introducing a managed and automated pipeline with proven best practices that help bring the feature to production efficiently.

In the event of production, the main benefit of CI/CD is that we can roll back to a previous working version of our product and fix the current patch separately. This benefit lessens the system downtime and increases compliance and confidence.

What is CI/CD Security?

While automation and team-level collaboration are two of the most crucial benefits of adopting a CI/CD practice, administering security in such frameworks is often more complex than doing so on legacy models. CI/CD security is the ideology of bringing security into DevOps, securing the pipeline, and streamlining a workflow that meets compliance standards.

This is in line with the “shift-left” testing philosophy. In other words, it’s better to detect and prevent security issues as early in the development process as possible.

But why is this important? Most of the time, when an application reaches the production stage, it has already been through multiple rounds of development, testing, and refinement. By the time it gets to prod, the code is pretty stable. So if a security issue does arise at this stage, it’s usually because of a misconfiguration or an external threat.

With CI/CD security in place, on the other hand, any potential security vulnerabilities can be detected and corrected much earlier in the development process – before they have a chance to cause any damage. DevOps pipelines tend to introduce code repositories and container image registries that can be private or public. This can bring in misconfigurations or additional open source vulnerabilities. CI/CD security is the best way to tackle these issues at the root.

Common Security CI/CD Pipeline Threats

Poisoned Pipeline Execution (PPE)

Poisoned pipeline execution (PPE) is a type of software vulnerability that can be exploited by attackers to inject malicious code into a running process. The vulnerability can be exploited to take control of the process or to cause the process to crash. PPE vulnerabilities are often found in both client-side and server-side applications that accept untrusted input without appropriate data sanitation.

An effective approach to prevent PPE vulnerabilities is to avoid using untrusted input in server-side applications and to avoid using untrusted output (generated from the server) in client-side applications. As a recommended practice, all untrusted input should be carefully validated to ensure that it does not contain any malicious code. As for untrusted outputs, the data should be displayed as untrusted and should be used with caution.

Although poisoned pipeline execution vulnerabilities can be difficult to exploit, they pose severe consequences if exploited successfully.

Insufficient Pipeline-Based Access Controls

When distributed teams develop or make changes to one part of the module, unified access unintentionally changes the state of other dependent sections. In such instances, setting unified access to pipelines is not recommended and can become daunting very quickly as CI/CD is all about collaboration from diverse teams.

Insufficient access controls can be detrimental to security as they can lead to privilege escalation and data leakage. An attacker with low privileges may be able to gain high-privileged access or modify code that is not supposed to be accessible. Also, if there are no access controls in place, an attacker may be able to view or tamper with data that is not intended for them.

Ungoverned Use of Third-Party Services

A common CI/CD security threat arises through the ungoverned use of third-party services. Although this offers developers the freedom to choose the services they intend to use, this can introduce a number of security issues, as the security risks associated with using these services are either unknown or ignored.

For example, a developer may choose to use a public code repository and container image registry that is not owned or controlled by their company. This code repository may not have the same security controls in place as the company’s internal code repositories. As a result, the code that is stored in this public repository may be less secure and more vulnerable to attack.

To help mitigate these risks, companies should adopt policies and procedures governing the use of third-party services. Enforcing such policies should ensure that only approved services that satisfy the organization’s security controls are used and are updated regularly to tackle the changing threat landscape.

Dependency Chain Abuse

Another common CI/CD security issue arises from the abuse of dependency chains, which represent a series of dependencies between components of an application stack.

For example, a web application may depend on a number of other components, such as a database, an application server, and a web server. Attackers may exploit a vulnerability in the database that allows them to execute arbitrary code on the database server. They can then use this code execution to gain access to the web application and the application server. This type of attack is known as a “dependency chain attack“.

To help mitigate dependency-based risks, organizations should carefully consider and manifest dependencies between components in their systems. They should also put in place policies and procedures to ensure that these dependencies are maintained and monitored to prevent vulnerabilities.

Best Practices for CI/CD Security

Embedding Security Gates

Security gates can be embedded deeply in both CI and CD pipelines. Their purpose is to detect security issues early and prevent them before they reach production, such as documented vulnerabilities, common misconfigurations, permissions, and others. CI/CD security gates include not only the technical controls that need to be applied but also the way we think about security posture, risk appetite, and how to continuously enforce cloud-native security in an ongoing manner.

Avoid hardcoding secrets

Ad-hoc development and testing with hardcoded values is attractive and gets the job done on the go, but placing keys and tokens in the code can trigger a security compliance issue. Avoiding this at all costs is ideal.

The recommended practice is to use environment variables, which are variables that are set outside of the code and can be used by the code at runtime. This way, secrets are not hardcoded in the code and can be easily changed without having to modify the code itself.

Isolate configurations from dev, production, and non-production

Configurations sometimes get executed irrespective of the source and permissions, producing unclear error stack traces and making debugging a nightmare.

The recommended approach is to place config variables within different files using use-case-specific naming conventions to avoid conflicts and crashes in both non-prod and prod environments.

Implement explicit and private/protected variables

As global and implicit values in the code complicate refactoring and debugging, a recommended practice is to use explicit, private, and immutable variables. The practice increases the stateful and stateless guarantees with an added advantage in developing performant and secure code.

Restrict pipeline access with dedicated policies

As most CI/CD pipelines are deployed and reside in the cloud, interacting with external services and platforms is typically achieved via REST API. Allowing infrastructure resources to open public connections and access to all user groups can however lead to data exposure and GDPR compliance.

To prevent this, it is recommended to restrict security policies and resource access to limited groups for enhanced security and trust.

Implement logging and tracking

It is crucial to keep a tab on what went wrong and which part of the workflow is susceptible to attack vectors. Setting up logging at different levels and declaring paths for log collection during the initial stages of development helps keep a track of the execution of pipeline stages.

The Different Layers of CI/CD Security

CI/CD security comprises multiple comprehensive and multi-layered approaches to toughen pipeline security at every stage. The layers are listed as follows:

Runtime Security

Bugs and misconfigurations are easy to bypass into production. The runtime security layer plays a vital role in scanning and detecting security leaks and threats from live applications running in production through a monitoring standpoint.

Static Security Testing

Before the deployment stage, the code developed by internal teams has to undergo pre-defined tests for identifying potential vulnerabilities. The static security testing layer is about creating test cases in version control systems for detecting vulnerabilities before merging the features into the main branch.

Vulnerability Scanning

Extend test cases of automated pipelines by adding application and environment scans to detect and classify weak code, infrastructural gaps, and third-party service resource leaking.

A comprehensive vulnerability scanning also helps maintain a central repository with known vulnerabilities. Scanning the pipeline helps detect and compare classified issues with the scanned changes in the merged repo for remediating vulnerabilities.

Misconfiguration Scanning

One of the most important things you can do is keep your systems up-to-date and free from any vulnerabilities. This process is known as misconfiguration scanning, and it involves regularly scanning your system for any potential security issues.

One of the most common ways that hackers gain access to systems is through vulnerabilities that exist due to poor configuration. By regularly scanning your system, you can ensure that any potential issues are found and fixed before they can be exploited by criminals.

In addition to keeping your system secure, misconfiguration scanning can also help improve its performance. By ensuring that all of your system’s components are properly configured, you can help eliminate any bottlenecks or other issues that can slow down your system.

Auditing and Monitoring

Logging phases of pipeline execution and tracking the success and failure patterns can be a resource in analyzing defect patterns and vulnerability detection for the overall CI/CD pipeline’s security enhancement.

Periodic auditing can help tackle evolving threats because it gives an overview of the system’s health and resource utilization. The security team can use this data to further fine-tune detection mechanisms and response processes.

Conclusion

Continuous integration and delivery pipelines have become integral to DevOps processes. A paradigm shift is underway that emphasizes automating the entire process of code development, testing, and deployment while ensuring compliance with security best practices.

As more and more businesses move towards a fully automated CI/CD culture, the challenge for enterprises will be to evolve their security practices to meet the needs of a fast-paced, continuous delivery model.

In this post, we discussed the importance of CI/CD security for DevOps processes and looked at the different layers of security that need to be considered for a robust pipeline. We also looked at some possible ways to secure and protect your pipeline from potential attacks.

The post CI/CD Security: How to Keep your CI/CD Pipelines Secure appeared first on ARMO.

Kubernetes version 1.25 – everything you should know

Jonathan Kaftzan — Mon, 22 Aug 2022 10:44:23 +0000

Kubernetes’ new version – version 1.25 – will be released on Tuesday 23rd August 2022, and it comes with 40 new enhancements in various areas and numerous bug fixes.

This blog will focus on the highlighted changes from each special interest group (SIG) in the upcoming release and ensure you are confident before upgrading your clusters.

API Machinery

There are two new and shiny enhancements from the API machinery group.

CRD Validation Expression Language (graduation to beta)

Custom resources are the key extension point in Kubernetes to create and manage new resources in the Kubernetes API. With the upcoming release, CRD validation using expression language graduates to beta. Instead of deploying and using webhooks for validation, it will now be possible to add validation rules to the CRD schema and manage them side by side with the resource specification.

Retriable and Non-Retriable Pod Failures for Jobs (alpha release)

Job resources are the way to run one-time tasks in Kubernetes. However, the job API in Kubernetes is minimal regarding failure handling. With this new alpha feature, there’s a new field in podFailurePolicy in the job specification. You can define rules as follows, an example job spec with a failure policy from Github/Kubernetes, and take action on the outcome of the container:

apiVersion: v1
kind: Job
spec:
  template:
    spec:
      containers:
      - name: main-job-container
        image: job-image
        command: ["./program"]
      - name: monitoring-job-container
        image: job-monitoring
        command: ["./monitoring"]
  backoffLimit: 3
  podFailurePolicy:
    rules:
    - action: Terminate
      onExitCodes:
        containerName: main-job-container
        operator: In
        values: [1,2,3]
    - action: Ignore
      onPodConditions:
      - type: DisruptionTarget

Apps

The Apps SIG focuses on deploying and managing complex applications in Kubernetes. In the 1.25 release, there are two crucial enhancements in this area.

Add minReadySeconds to StatefulSets (graduation to stable)

minReadySeconds is a new—but stable—field in StatefulSet resources to ensure the workload is ready after the pods become available. These extra buffer seconds are beneficial when containers start, but it takes time to be ready for the application to accept requests.

TimeZone Support in CronJob (graduation to beta)

CronJob instances are created by the schedule provided in the resource specification. However, the time zone of the newly created resources is based on where the controller-manager is running. With the new enhancement, you get a new field, spec.timeZone, where you can use a valid timezone from the tz database.

Auth

Here, we have one critical depreciation and one new alpha release from the authorization, authentication, and cluster security policy area.

Removal of PodSecurityPolicy

In Kubernetes 1.25, PodSecurityPolicy is completely removed after its depreciation in the 1.21 version. PodSecurityPolicy was the solution to define rules on a pod’s capabilities, but it became complex and confusing over time. Instead, Kubernetes has now implemented Pod Security Admission controllers with a clear migration path.

KMS v2 Improvements (alpha release)

Kubernetes stores all of its data in etcd, and it is not encrypted by default. Because of this, Kubernetes offers external mechanisms like Key Management Service (KMS) providers to safely store data in etcd. The new v2alpha1 enhancement focuses on making KMS handle key rotation automatically. In addition, it offers improvements on KMS plugin health checks and observability of the operations between the API server and KMS.

Network

There are two graduations from the networking area in the upcoming release.

NetworkPolicy Port Range (graduation to stable)

In ingress and egress network policies, you need to specify each port one by one with the current Kubernetes API. The new—and now stable—feature adds a field named endPort to easily declare a port range. For instance, you can apply a rule from port 32000 to 32768 as follows:

spec:
  egress:
  - ports:
    - protocol: TCP
      port: 32000
 endPort: 32768

Reserve Service IP Ranges for Dynamic and Static IP Allocation (graduation to beta)

Kubernetes service resources expose applications inside and outside the cluster. There are two methods to choose an IP for a service resource: Either Kubernetes assigns a random IP from a configured range, or the user statically specifies an IP from the same range. You can divide the IP range using the ServiceIPStaticSubrange field, which has graduated to beta, and avoid collisions while assigning IP addresses to services in Kubernetes.

Node

In the 1.25 release, there are three general availability (GA) graduations, as well as one beta and one alpha release, in the node area.

Ephemeral Containers (graduation to stable)

Debugging a distributed living system is always challenging since it is not easy to connect, send requests, and check the results. With ephemeral containers, you can add a container to a running pod. Since the application container images are minimal without any shell, curl, or debugging tool, ephemeral containers are beneficial for quickly spinning a debugger container.

For instance, you can attach an interactive ephemeral busybox image to db-pod with the following command and start debugging:

$ kubectl debug db-pod -it --image=busybox
Defaulting debug container name to debugger-8xzrl.
If you don't see a command prompt, try pressing enter.
/ #

cgroups v2 (graduation to stable)

cgroups is one of the key Linux kernel functionalities to organize and manage container resources on nodes. In the early days of Kubernetes, all container runtimes were built using cgroup v1, but now cgroups v2 support has graduated to general availability. With cgroups v2, container workloads will work more securely, including rootless containers, and more reliably with the latest kernel functionalities.

Add Configurable Grace Period to Probes (graduation to stable)

There is a new—and now stable in liveness probes—field called terminationGracePeriodSeconds in addition to the terminationGracePeriodSeconds on the pod level. The separation of these fields helps decide how long Kubernetes will wait to kill a container under a normal shutdown and due to a failed liveness probe.

seccomp by Default (graduation to beta)

Kubernetes allows increasing container security by defining seccomp profiles; it has been an alpha feature since the 1.22 release. Enabling Seccomp by default adds a security layer to prevent CVEs and 0-days, and now this feature has graduated to beta in the 1.25 release.

Add CPUManager Policy Option to Align CPUs by Socket Instead of NUMA Node (alpha release)

With the new CPU architectures, there’s an increase in the number of NUMA (non-uniform memory access) nodes per socket. The new alpha feature adds a new CPUManager policy option as align-by-socket. With this, CPUs will be considered aligned at socket boundaries instead of NUMA boundaries.

Security

Version 1.25 has a single critical enhancement from the security area.

Auto-Refreshing Official CVE Feed (alpha release)

Kubernetes is one of the most active open-source repositories and thus has many issues and PRs, which, in relation to CVEs, are impossible to filter. The new alpha feature ensures the labeling of issues and PRs with the help of automation. This new approach will let you list CVEs with the relevant information as an end-users, maintainers, or platform providers.

Scheduling

You’ll get one new alpha release from the scheduling area.

Respect PodTopologySpread After Rolling Upgrades (alpha release)

PodTopologySpread is a part of the pod API to define constraints on how pods are distributed over the cluster, such as per region, zone, node, or any other user-defined topology. For instance, let’s assume you have a 20-node cluster and an auto-scaling application with a minimum of 2 and a maximum of 15. When a minimum of 2 instances are running, you would not want both of them to run on the same node—or availability zone. These constraints are helpful, as they increase availability in case of failures in the cluster. With the 1.25 release, Kubernetes will also respect the spread constraints in the rolling-upgrade stage.

Storage

From the storage area, there are two essential general availability graduations and one alpha release.

Local Ephemeral Storage Capacity Isolation (graduation to stable)

Pods use temporary storage to write their logs and emptyDir mounts and as a cache. Without any isolation, every pod on the node shares the same temporary storage pool on a “best-effort” basis. In other words, pods do not know how much space is allocated to them or left on the node. With the storage capacity isolation feature, which will be generally available in the upcoming release, pods can reserve their own storage from the ephemeral pool.

In-Tree Storage Plugin to CSI Driver Migration (graduation to stable)

Migration of the in-tree plugins to external CSI plugins graduates to stable in version 1.25. This is an important step that includes the removal and depreciation of many volume plugins:

Depreciation: GlusterFS, Portworx
Removal: The Flocker, Quobyte, and StorageOS
Migration to CSI plugin: AWS EBS, GCE PD, vSphere

Retroactive Default StorageClass Assignment (alpha release)

The default storage class is configured mainly during cluster creation by the cluster admin. However, when there are changes in the underlying storage provider or business requirements, you should also change the default storage class in the cluster. The new alpha feature focuses on changing the Kubernetes behavior to be retroactive for PVCs without any storage class.

Conclusion

Kubernetes 1.25 aims to make Kubernetes more secure, reliable, and flexible. Ensure that you are well-equipped for the latest changes in the release and upgrade your infrastructure promptly. Check the Kubernetes blog and release notes to learn more about the enhancements and the latest changes.

The post Kubernetes version 1.25 – everything you should know appeared first on ARMO.

The New Kubernetes Gateway API and Its Use Cases

Jonathan Kaftzan — Mon, 25 Jul 2022 08:46:36 +0000

Despite being a large open-source and complex project, Kubernetes keeps on evolving at an impressive pace. Being at the center of various platforms and solutions, the biggest challenge for the Kubernetes project is to remain vendor-neutral. This is the reason the community has come up with Kubernetes Gateway API.

Gateway API is a new project by the SIG-Network community designed to enhance service networking through consistent and extensible interfaces that multiple vendors can implement to offer broader choices to the development community.

In this article, we’ll look into Gateway API, how it is different from Ingress, and what we can do with it.

How Gateway API Differs from Ingress

Usually, after deploying a Kubernetes application, you need to expose it to the end users. This is typically done using Ingress Controller constructs for north-south traffic. The Ingress API object defines the routes and mapping of the external traffic to Kubernetes services; it also offers load balancing, SSL termination, and name-based virtual hosting.

Many off-the-shelf controllers such as NGINX Ingress controller and HAProxy implement the Ingress interface. These controllers have differentiated themselves from each other by providing more features such as advanced load balancing via proprietary extensions. The current Ingress API scope is very limited to ensure better portability in the ecosystem.

Gateway API is an evolution of Ingress that provides advanced features natively by extending the API definitions. Some of these features were provided by individual Ingress vendors as private extensions, but these implementations were not aligned with each other. Now, with Gateway API, many such capabilities will be implemented by multiple vendors according to a single specification so that users will have a choice of different implementations.

Important additions by Gateway API include HTTP and TCP routes, traffic splitting, and a role-oriented approach that allows cluster admins and developers to focus on the setting relevant to their responsibilities.

Kubernetes Gateway API Concepts

The SIG-Network community has come up with the following design goals for Gateway API to improve upon the Ingress resource:

Role-oriented: API resources to manage the Kubernetes service networking should model the different organizational roles that handle their scope of resources, as shown below:

Portable: Like Ingress, these APIs should also be portable and have a universal specification.
Expressive: API natively supports core functionalities such as header-based routing, traffic weighting, and other advanced features that were only possible in Ingress through custom annotations.
Extensible: Every layer of the API is extensible via custom resources, enabling granular customization of the API’s structure.

Other notable capabilities of Kubernetes API Gateway are:

GatewayClasses: These formalize the implementation types for load balancing, allowing users to clearly grasp the various capabilities of the Kubernetes resource model.
Shared gateways and cross-namespace support: These allow you to create separation among teams per their responsibilities. We will discuss this further in the next section.
Typed routes and typed backends: These offer support for HTTPRoute, TCPRoute, TLS, UDPRoute, etc. to provide coverage for all protocols.

Now, the question is, what can you do with Gateway API?

Advanced Traffic Routing and Progressive Delivery

One of the most important features missing in Ingress is advanced traffic routing. Up until now, this was resolved by way of a service mesh, which made it complex and tightly coupled with the mesh implementation.

Below, we cover some of the advanced routing scenarios that Gateway API provides natively so you no longer need a mesh.

Simple Gateway

This is the simplest way to expose the service using the Gateway and Route, managed by the same owner. The load balancer is managed by the Gateway controller, which sends all arriving traffic coming to the service. This gives autonomy to the service owners and allows them to expose the service as they want, a model that is similar to Ingress:

Figure 2: Simple Gateway pattern where traffic flows via a load balancer. (Source: Kubernetes Gateway API)

HTTP Routing

HTTPRoute allows you to route the traffic to multiple services based on filters. In the example below, there are two routes attached to the same gateway.

HTTPRoute foo-route where all the traffic that matches the hostname foo.example.com and /login is sent to foo-svc.
HTTPRoute bar-route where all the traffic from host bar.example.com/* is sent to bar-svc, and traffic with the header env: canary is sent to the bar-svc-canary service.

This model allows you to introduce a progressive delivery strategy:

Figure 3: HTTP routing with multiple paths and header-based routing. (Source: Kubernetes Gateway API )

HTTP Traffic Splitting

In this model, you can have weighted traffic routing, which you can combine with A/B or canary strategies to achieve complex rollouts in a simple way. In the above diagram, the HTTPRoute is splitting the traffic 90/10 and sending it to the foo-v1 service and foo-v2 service, respectively.

Here, the weight is not in the percentage. Instead, the sum of all weights within the single route is the denominator of all the backends. This allows for a gradual rollout of the newer version:

Figure 4: HTTP traffic splitting example (Source: Kubernetes Gateway API)

You can also achieve header-based routing by having the same weight but a filter on the header. A canary route with header traffic=test is sent to the new service foo-v2 for testing, thus allowing for safer releases without impacting v1 traffic:

Figure 5: Header-based routing (Source: Kubernetes Gateway API)

Cross-Namespace Routing

In the cross-namespace routing, you can share the gateway(s) and have team-based control on their routes. For example, in a large setup, an infrastructure team may be responsible for managing the shared gateway, while individual teams manage their routes. This allows for clear separation of duties among the users:

Figure 6: Cross-namespace routing using a shared gateway model (Source: Kubernetes Gateway API)

In the table below, we can see the different roles and responsibilities of the owner for each resource type. For example, an infrastructure provider may have access to all the layers, but an application developer is limited to owning only routes to their service:

| | GatewayClass | Gateway | Route |
| Infrastructure Provider | Yes | Yes | Yes |
| Cluster Operator | Sometimes | Yes | Yes |
| Application Admins | No | In specific namespaces | In specific namespaces |
| Application Developers | No | No | In specific namespaces |

TLS

Gateway API supports TLS configuration at various points in the network path between the client and service—for upstream and downstream independently. Depending on the listener configuration, various TLS modes and route types are possible; support for cert-manager integration is also available:

Figure 7: TLS configuration for both upstream and downstream (Source: Kubernetes Gateway API)

The following table from the official documentation provides clarification as to when an object should be used and the TLS support provided by each:

TCP Routing

You can implement numerous protocols with Gateway API, including support for TCPRoute. The listeners under the gateway should have protocol: TCP in the configuration to enable TCP routing and allow you to manage TCP traffic.

Integration with Progressive Delivery Tools

Combined with the various advanced traffic routing options, API Gateway currently provides integration with Flagger—a progressive delivery tool for advanced deployment strategies such as A/B, blue-green, and canary. It works with all implementations of Gateway API.

Portability

The main idea of introducing Gateway API was to bring consistency across the Ingress capabilities offered by various solutions. This also makes it portable, so when you move the workload to a different provider or write a multi-cloud solution, it will work the same way without requiring a significant amount of change in the specifications.

Kubernetes Gateway API Demo

There are various vendors working to introduce Gateway API implementations such as Contour, Emissary-Ingress, GKE, and Traefik. You can get the list of implementations here. Just keep in mind that most of them are in beta and not really production-ready.

Conclusion

Kubernetes Gateway API has evolved to provide expressive, portable, and extensible API specs for implementers such as infrastructure providers, cluster operators, and application developers. Although not a complete replacement for Ingress as of now, you should aim to use Gateway API wherever possible, as it does provide more options for developers without introducing a lot of annotations or non-portable changes.

Furthermore, all the projects in the service mesh and Ingress controller space have implemented Gateway API along with tools like Flagger and cert-manager—and you should expect more to jump on board soon as it becomes more popular in the CNCF ecosystem. Most of the implementations discussed above are in beta but are expected to be GA soon. To learn more about Gateway API’s graduation to beta and plans for its GA release, check out this talk at KubeCon + CloudNativeCon Europe 2022.

The post The New Kubernetes Gateway API and Its Use Cases appeared first on ARMO.