Forem: Gadi Naor

New Kubernetes Node Vulnerability (CVE-2020-8558) bypasses localhost boundary

Gadi Naor — Wed, 22 Jul 2020 08:11:30 +0000

Vulnerability Description and Impact

A security issue was discovered in kube-proxy which allows adjacent nodes/hosts to reach TCP and UDP services bound to 127.0.0.1 running on the node or in the node's network namespace (host network). This breaks security assumptions made by services listening on localhost.

This security bug was originally raised in issue #90259 which details how the kube-proxy sets net.ipv4.conf.all.route_localnet=1 and causes the system not to reject traffic to localhost which originates on other hosts (martian traffic). Such traffic would look like packets on the wire with an IPv4 destination in the range 127.0.0.0/8 and a layer-2 destination MAC address of a target node may indicate that an attack is targeting this vulnerability.

For example, if a cluster administrator runs a TCP service on a node that listens on 127.0.0.1:1234, because of this security bug, that service would be potentially reachable by other hosts on the same LAN as the node, or by containers running on the same node as the service. If the example service on port 1234 required no additional authentication (because it assumed that only other localhost processes could reach it), then it could be vulnerable to attacks that make use of this security bug.

While many Kubernetes installers explicitly disable the API Server's insecure port, and Kubernetes v1.20 is planned to remove this insecure option, An API server that uses this insecure option and listens on 127.0.0.1:8080 will accept requests without authentication.
To mount such an attack on the API server, an attacker must have access to another system on the same LAN or with control of a container running on the master. Managed Kubernetes services such as EKS, AKS, GKE and others should be resilient attacks on the API server insecure port.

Are You Vulnerable?

The vulnerability affects kubelet & kube-proxy which are core Node components:

Affected Versions:

kubelet/kube-proxy v1.18.0-1.18.3
kubelet/kube-proxy v1.17.0-1.17.6
kubelet/kube-proxy <=1.16.10

Or if one or more of the following items are applicable to your environments:

Your cluster nodes run in an environment where untrusted hosts share the same layer 2 domain (i.e. same LAN) as the cluster nodes.
Your cluster allows untrusted pods to run containers with CAP_NET_RAW which is enabled by default by Kubernetes.
Your nodes (or hostnetwork pods) run any localhost-only services which do not require any further authentication. To list services that are potentially affected, run the following commands on nodes:

lsof +c 15 -P -n -i4TCP@127.0.0.1 -sTCP:LISTEN
lsof +c 15 -P -n -i4UDP@127.0.0.1

Risk:

Typical Clusters: medium (5.4) CVSS:3.1/AV:A/AC:L/PR:N/UI:N/S:U/C:L/I:L/A:N
In clusters where API server insecure port has not been disabled: high (8.8) CVSS:3.1/AV:A/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H

Automatic Detection of adjacent Node attacks with Alcide Runtime

This new vulnerability has long been covered by Alcide Runtime without requiring any configuration by the user.
This detection has been part of Alcide Runtime for a long time, even prior to this vulnerability’s disclosure, and there was no need for us to add additional detection capabilities.

The traffic is flagged as spoofed traffic because localhost traffic should never cross network boundaries. Furthermore, if you explicitly define firewall policies using Alcide’s microservices firewall, then pods can’t access other resources in the network unless explicitly allowed. This is what zero-trust networking is all about!

Automatic Detection of API Server attacks with Alcide kAudit

Exploit attempts of this vulnerability via the unsecured port of the API Server would show up in its audit log as entries from the "system:unsecured" principal, similar to entries from K8s services on the Master node accessing the API Server locally.

Alcide kAudit analyzes the audit log of the Kubernetes API Server, continuously updates and compares behavioral activity profiles to actual observed activity and automatically detects anomalous access patterns from cluster principals (in this case, "system:unsecured" account) to various resource types, k8s APIs, namespaces, and specific resources. Thus, Alcide kAudit can automatically alert security teams to suspected attacks that rely on CVE-2020-8558 as they occur.

Furthermore, Alcide kAudit users can configure and customize it to alert on audit entries violating their specific policy. They can, for example, add alerts on API Server activity that reads or modifies sensitive namespaces or resources in the cluster, and use such alerts if they happen as an indicator that an exploit may be in progress and should be investigated.

Automatic Detection of Vulnerable Clusters with Alcide Advisor

Alcide Advisor is a Kubernetes multi-cluster vulnerability scanner that covers rich Kubernetes and Istio security best practices and compliance checks such as Kubernetes vulnerability scanning, hunting misplaced secrets, or excessive secret access, and many more security configuration and compliance checks.

With Alcide Advisor users can:

Identify the vulnerable clusters for this vulnerability (as well as other CVEs)
Identify and explicitly allow/deny which Pods can run with elevated privileges that enables CAP_NET_RAW
Identify and explicitly allow/deny which Pods can run on the host network.
On applicable environments, identify Pods that can run on master nodes that can potentially exploit the Kubernetes API Server.

Conclusion

Kubernetes, like any software, has bugs and vulnerabilities. Leveraging Kubernetes as cloud-native application infrastructure requires operators to monitor and secure all the moving parts, whether these are the application workloads or the platform and infrastructure components. CVE-2020-8558 joins other recent vulnerability disclosures (CVE-2020-8555 and CVE-2020-10749) and highlights the need for a purpose-built Kubernetes security solution that can drive cluster operators to run workloads, applications, and infrastructure while leveraging the best security practices of the native Kubernetes security controls, as well as security monitoring & prevention.

Start your 14-day trial or request a demo.

Helm Scan With GitHub Actions & K8s Advisor

Gadi Naor — Tue, 21 Apr 2020 07:21:32 +0000

GitHub Actions is a recent continuous integration (CI) and continuous deployment (CD) service from GitHub. GitHub Actions powers GitHub's built-in continuous integration service. In its essence, GitHub Actions help developers automate software development workflows in the same place they store code and collaborate on pull requests and issues.

GitHub Actions enable developers to write individual tasks, called actions, and combine them to create a custom workflow. Workflows are custom automated processes that developers can set up in their repository to build, test, package, release, or deploy any code project on GitHub.

Create Kubernetes KIND Cluster

Kubernetes IN Docker, KIND, is a tool to create local clusters for testing Kubernetes using Docker containers. KIND was primarily designed for testing Kubernetes itself but may be used for local development or CI.

Multi-node clusters and other advanced features may be configured with a config file - the detailed usage information and documentation are here. GitHub Actions has a marketplace for reusable actions, and the folks behind Helm already managed to put together Kind Cluster, a reusable action that can be plugged into GitHub’s automation workflow.

Security Scanning of Helm Charts with GitHub Actions Workflow

Alcide Advisor, an API driven, Kubernetes security and hygiene scanner, has a wide integration surface into the continuous deployment (CD) platforms. However, GitHub's actions combined with KIND, introduce an interesting approach for scanning helm charts in continuous integration (CI) stage.
In the example below, a GitHub workflow has 3 sequential jobs:

Build
Test
Advisor Scan

The Advisor Scan Job performs the following steps:

Download Helm 3
Launch Kind Cluster using a GitHub Action
Install a chart (uswitch kiam in this example) into a specific namespace
Download Alcide Advisor scanner
Scan with Alcide Advisor the namespace into which kiam was installed
Publish the scan report into the pipeline artifacts

Conclusion

Helm is the de-facto tool for collaborating when creating, installing, and managing applications inside of Kubernetes. Rendering helm charts with configuration into a cluster that can be scanned by Alcide Advisor, opens the door for developers & DevOps to ‘get a handle‘ on the security and hygiene level of new helm charts as well as helm charts changes. To see the full pipeline example go to https://github.com/alcideio/pipeline

GitOps - A Security Perspective (Part 1)

Gadi Naor — Mon, 13 Apr 2020 13:03:45 +0000

GitOps is a paradigm that puts Git at the heart of building and operating cloud native applications by using Git as the single source of truth and empowers developers to perform what used to fall under IT operations. This post is part a blog post series covering GitOps and Kubernetes security.

Kubernetes - A GitOps Companion

Kubernetes, as the new application server, leverages a “declarative” approach when it comes to building cloud native application, which means that application configuration is guaranteed by a set of facts instead of by a set of instructions. With application’s declarations versioned in Git, we have a single source of truth, our apps can be easily deployed and rolled back to and from Kubernetes, and when disaster strikes, your cluster’s infrastructure can also be reproduced.

With Git at the center of the delivery pipelines, developers can make pull requests to accelerate and simplify application deployments and operations tasks to Kubernetes.

Is GitOps right for you?

The fresh approach that GitOps + Kubernetes brings into the application delivery lifecycle is undeniably different, increasing engineering velocity, as well as simplifies building the CI+CD pipelines themselves. The question of whether a GitOps ‘Pull’ approach is a better fit than ‘Push’ approach is really a matter of engineering & operational culture of an organization, as well as almost a theological question of whether engineering are accountable for security and what visibility into this application server blackbox, security teams require.

GitOps & Security

GitOps changes are synched into the cluster only through the cluster git repository users. The repository is secured at the same level of the git user accounts. A compromised user account in with permissions to push into the cluster git repo can introduce changes that will result in a data breach, service disruption or anything in between. There must be additional guardrails that GitOps infrastructure must implement in the form of whitelisting in order to have a cluster side guardrails.
GitOps tools like Flux, ArgoCD and alike are practically running with cluster god permissions - and are persistent in the cluster. Kubernetes Dashboard, which is considered to be a high risk cluster, is often times removed from the cluster.
‘Push’ based CD pipelines, such as Spinnaker, Jenkins and alike are external to the cluster, invoke on-demand, and introduce automation-driven changes into the cluster.

Example: Flux RBAC Permission - Cluster God * * * * *
GitOps tools like Flux, ArgoCD and alike require cluster external access, represented as domain names (github.com, bitbucket.org, gitlab.com,..) which means that the Kubernetes native policies are not suitable to implement to segment those highly privileged in-cluster components.
Application secrets in a GitOps era requires a Kubernetes external secret provider. For example Hashicorp Vault, AWS KMS, Azure Vault and alike. Alternatively, teams can revert into Git Secrets which means committing secrets into git in their encrypted form, and decrypting the secrets before application consumption.

Conclusion

Continuous integration/continuous development (CI/CD) with the Kubernetes ecosystem does have a variety of tools to choose from and organizations should use the tools that are best suited for their specific use cases and culture. Glueing all the pieces together is not trivial. Integrating security and consuming security insights by various stakeholders is an equally challenging task to achieve. GitOps simplifies this in some aspects, but complicates in other aspects.

Stay tuned for Part 2.

Join our upcoming webinar on April 22nd: GitOps Best Practices for Continuous Deployment and Progressive Security.

Kubernetes RBAC Visualization

Gadi Naor — Sun, 05 Apr 2020 15:16:19 +0000

Role-based access control (RBAC) is a method of regulating access to a computer or network resources based on the roles of individual users within your organization. RBAC authorization uses the rbac.authorization.k8s.io API group to drive authorization decisions, allowing you to dynamically configure policies through the Kubernetes API.

Permissions are purely additive and there are no “deny” rules.

Define permissions on namespaced resources and be granted within individual namespaces
Define permissions on namespaced resources and be granted across all namespaces
Define permissions on cluster-scoped resources

Roles are used to define API access rules for resources within the namespace of the role, and ClusterRole is used to define API access across all cluster namespaces.

Alcide’s rbac-tool viz

Alcide’s rbac-tool, an open-source tool from Alcide, introduces a visualization functionality of the relationships between the resources that make your cluster RBAC configuration.

The diagram above captures the various relationship combinations between resources.

Roles - Defines the policy rules that constitute which API actions (read/create/update/delete) the subject (user/service) is allowed to perform on resources within the namespace resources.

ClusterRoles - Defines the policy rules that constitute which API actions (read/create/update/delete) the subject (user/service) is allowed to perform on resources cluster-wide.

Bindings, are the Kubernetes RBAC resources that define the link between principals (users of automated services).
Bindings can point to multiple Roles:

RoleBindings can point to ClusterRoles which grants the subject (user/service) cluster-wide access to the resources specified in the rules.

ClusterRoleBindings can point to ClusterRoles which grants the subject (user/service) cluster-wide access to the resources specified in the rules.

Nginx Ingress Controller RBAC

The following diagram shows the moving parts of the RBAC resources created by an Nginx Ingress Controller.

You can see 2 roles were created:

A role that defines the allowed resources access within the namespace
ClusterRole defines the cluster-wide access permissions

Note for example that the ClusterRole grants Nginx-ingress account to
Update the resource status of ingress within the extensions and networking.k8s.io API group.

The above visualization was generated by running the following command:

$ rbac-tool viz --include-subjects="nginx-ingress"

Under the hood, Alcide’s rbac-tool connects to the cluster context pointed by your kubeconfig , lists the various RBAC related resources, and visualize the resources based on the command line filters.

Example: API access for system:unauthenticated Group on GKE

$ rbac-tool viz --include-subjects="system:unauthenticated"

Example: GCP permission covered cloud-provider ServiceAccount for GKE

$ rbac-tool viz --include-subjects="^cloud-provider" --exclude-namespaces=""

Conclusion

Kubernetes RBAC is a critical component in your Kubernetes deployment, which is definitely something cluster operators and builders must master.
Alcide’s rbac-tool visualization and filtering capabilities helps to unfold and simplify Kubernetes RBAC.

Kubernetes RBAC | Moving from ‘It's Complicated’ to ‘In a Relationship’

Gadi Naor — Wed, 01 Apr 2020 11:09:48 +0000

Role-based access control (RBAC) is a method of regulating access to computer or network resources based on the roles of individual users within your organization. RBAC authorization uses the rbac.authorization.k8s.io API group to drive authorization decisions, allowing you to dynamically configure policies through the Kubernetes API.

Permissions are purely additive and there are no “deny” rules.

A Role always sets permissions within a particular namespace ; when you create a Role, you have to specify the namespace it belongs in. ClusterRole, by contrast, is a non-namespaced resource, and grants access at the cluster level. ClusterRoles have several uses.
You can use a ClusterRole to:
define permissions on namespaced resources and be granted within individual namespace(s)
define permissions on namespaced resources and be granted across all namespaces
define permissions on cluster-scoped resources

If you want to define a role within a namespace, use a Role; if you want to define a role cluster-wide, use a ClusterRole.

Default Cluster Roles
While Kubernetes RBAC is a complex topic, one would always want to implement RBAC in the cluster. For this purpose Kubernetes offers out-of-the-box default cluster roles that can be used as a starting point.
These are visible in the output of kubectl get clusterrole, and four cluster roles you can use right away are:

cluster-admin
admin
edit
view

With these roles, you can start to define who can interact with your cluster and in what way. It is highly recommended to follow the principle of least privilege, and grant additional privileges as necessary for work to proceed.
Kubernetes RBAC Resource Relationship

Example: Nginx Ingress Controller RBAC Policy

Explicitly Tuning RBAC Policies - ‘It's Complicated’
Components like Operators or highly privileged controllers that require Cluster wide ‘Read-Only’ but do not necessarily require to read secrets for example, may leave a user to take a shortcut and provision RBAC policy with excessive permissions.

Kubernetes RBAC additive model does not enable us to implement semantics that capture:
deny access to specific resource groups,
allow access to all other resources.

If we can have the explicit set of policy access rules of all cluster resources - and “subtract” from that group the resources we would like to deny access to - we achieve the above semantics.

Let’s how we can achieve that:
Run kubectl api-resources and get all of the cluster installed/supported resources and their respective api-groups
Derive the RBAC policy from the above list
Manually tune the policy and reduce any resource and/or api-groups that we wish to deny access to.

While this method will work, it’s manual, and takes significant effort.
An easier way to achieve this is with Alcide rbac-tool

rbac-tool - Simplify RBAC Policy Tuning
Alcide rbac-tool has the ability to address this exact use case.
Example: Generate a Role policy that allows create,update,get,list (read/write) everything except secrets, services, networkpolicies in core,apps & networking.k8s.io API groups
rbac-tool gen --generated-type=Role --deny-resources=secrets.,services.,networkpolicies.networking.k8s.io --allowed-verbs=* --allowed-groups=,extensions,apps,networking.k8s.io

The generated policy is cluster specific.
For a Kubernetes KIND cluster v1.16 the generated policy looks as follows:

Conclusion
Kubernetes RBAC helps us as users, to define and regulate API access to the Kubernetes cluster. In many cases where users wish to achieve even more granular controls, Validating Admission Controllers are the Kubernetes construct to achieve that. The Kubernetes ecosystem, and open-source tools such as Alcide’s rbac-tool help to unfold and simplify Kubernetes RBAC. get it here: https://github.com/alcideio/rbac-tool

Unfolding & Sugar Coating for Kubernetes RBAC

Gadi Naor — Tue, 31 Mar 2020 17:16:02 +0000

Kubernetes RBAC

Role-based access control (RBAC) is a method of regulating access to computer or network resources based on the roles of individual users within your organization.
RBAC authorization uses the rbac.authorization.k8s.io API group to drive authorization decisions, allowing you to dynamically configure policies through the Kubernetes API.

Permissions are purely additive (there are no “deny” rules).

define permissions on namespaced resources and be granted within individual namespace(s)
define permissions on namespaced resources and be granted across all namespaces
define permissions on cluster-scoped resources

If you want to define a role within a namespace, use a Role; if you want to define a role cluster-wide, use a ClusterRole.

`rbac-tool viz`

A Kubernetes RBAC visualizer that generate a graph as dot file format or in HTML format.

By default 'rbac-tool viz' will connect to the local cluster (pointed by kubeconfig)
Create a RBAC graph of the actively running workload on all namespaces except kube-system

See run options on how to render specific namespaces, other clusters, etc.

Render Locally

rbac-tool viz --outformat dot && cat rbac.dot | dot -Tpng > rbac.png  && open rbac.png

Render Online

https://dreampuf.github.io/GraphvizOnline

Examples:

Scan the cluster pointed by the kubeconfig context 'myctx'

rbac-tool viz --cluster-context myctx

Scan and create a PNG image from the graph

rbac-tool viz --outformat dot --exclude-namespaces=soemns && cat rbac.dot | dot -Tpng > rbac.png && google-chrome rbac.png

Installation instructions can be found here: rbac-tool