Forem: Michael Chenetz

Understanding Kubernetes Authentication and Authorization

Michael Chenetz — Thu, 30 Nov 2023 15:24:21 +0000

Setting up user permissions properly in Kubernetes involves understanding and effectively using Kubernetes' Role-Based Access Control (RBAC) system. RBAC in Kubernetes allows you to regulate access to resources in the cluster based on the roles of individual users or groups of users. Here's a step-by-step guide to setting up user permissions:

1. Understand Kubernetes Authentication and Authorization

Firstly, understand that Kubernetes separates authentication (verifying who you are) from authorization (determining what you can do):

Authentication: You can authenticate users via certificates, tokens, basic auth, external identity providers like LDAP, etc.
Authorization: Once authenticated, authorization determines what actions the user can perform. Kubernetes uses RBAC for authorization.

2. Define Users and Groups

In Kubernetes, users are not created through the Kubernetes API but should be managed externally. For instance, you can have:

Service Accounts for processes in pods.
Normal Users managed externally (e.g., via an identity provider).
Groups, which are collections of users.

3. Use Role and ClusterRole

Role: A Role in Kubernetes is used to grant permissions within a specific namespace. It contains rules that represent a set of permissions.
ClusterRole: A ClusterRole is like a Role, but for the entire cluster. It's useful for granting permissions for non-namespaced resources (like nodes) or for namespaced resources across all namespaces.

4. Create RoleBindings and ClusterRoleBindings

RoleBinding: A RoleBinding grants the permissions defined in a Role to a user or set of users. It applies only within a specific namespace.
ClusterRoleBinding: A ClusterRoleBinding grants the permissions defined in a ClusterRole to a user or set of users cluster-wide.

5. Define Roles and Bindings

Create YAML files to define Roles/ClusterRoles and RoleBindings/ClusterRoleBindings. For example:

Role Example

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: default
  name: pod-reader
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "watch", "list"]

RoleBinding Example

kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: read-pods
  namespace: default
subjects:
- kind: User
  name: jane
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io

6. Apply the Configuration

Apply these configurations using kubectl apply -f <filename>.yaml.

7. Test the Permissions

Verify that the permissions are correctly set up by attempting to perform operations in the cluster as the specified users.

8. Regular Audits and Updates

Regularly audit and update the permissions to ensure they align with current operational requirements and security best practices.

9. Consider Using a Management Tool

For complex environments, consider using a Kubernetes management tool that offers a more user-friendly interface for managing roles and permissions.

Remember, proper setup of user permissions is crucial for the security of your Kubernetes cluster. Always follow the principle of least privilege, granting users only the permissions they need to perform their tasks.

What is Rego and how do you use it?

Michael Chenetz — Thu, 30 Nov 2023 15:20:39 +0000

The Rego language, used primarily with the Open Policy Agent (OPA), is a high-level declarative language for writing policy as code. Here's a basic illustration of how to use Rego:

Example Scenario: User Access Control

Suppose we have a system where we need to control user access based on their roles.

Data Model

First, define a simple data model. In a real-world scenario, this could be JSON data representing user roles and permissions:

{
  "users": {
    "alice": {"role": "admin"},
    "bob": {"role": "developer"},
    "eve": {"role": "intern"}
  }
}

Policy Definition

Next, write a Rego policy to specify who can access what. For instance, we might want only admins to access sensitive data:

package example

default allow = false

allow {
  input.user.role == "admin"
}

In this policy:

The package keyword defines a namespace (example).
default allow = false sets the default decision to deny access.
The allow rule permits access if the user's role is "admin".

Query

You'd then query this policy with input data to make access decisions. The input might look like:

{
  "user": {"role": "admin"}
}

You'd ask OPA a question like: "Given this input, should access be allowed?" If the input user role is "admin", the policy allows access, returning true.

Use in Code

In application code, you'd typically integrate OPA as a service or library. The application sends input data (e.g., user information) to OPA and gets back a decision based on your Rego policies.

This example is simplistic but illustrates the basic use of Rego. Real-world scenarios often involve more complex policies, multiple data sources, and integration with services like Kubernetes for dynamic policy enforcement.

Is your Cloud Native Application Secure?

Michael Chenetz — Fri, 17 Feb 2023 17:01:09 +0000

Description

The first thing we do when we learn a new technology is to get things up and running. There are so many things to learn and it's hard enough for us to just wrap our heads around the concepts. Add in all the complexity of cloud native and we can quickly see why there are about one million compromised Kubernetes clusters in the wild.

One of the biggest problems in this area is lack of education. It is assumed that Kubernetes is secure out of the box and this is just not true. This article will hopefully, help in highlighting some of the security vulnerabilities inherent in Kubernetes and how to remediate them.

Insecure by Default

Typical insecure default configurations of managed Kubernetes clusters include:
Using default account credentials to access the cluster API
Containers running unmasked, granting them complete access to a node’s procedures (/proc). This can allow malicious access to kernel parameters at runtime, or retrieval of sensitive information stored on the host.
Insecure volume configuration leading to leakage of information about other containers in the cluster

Below illustrates some of the components available to help you secure your cloud native app.

Security Context

The following is taken from the Kubernetes Docs
https://Kubernetes.io/docs/tasks/configure-pod-container/security-context/

Configure a Security Context for a Pod or Container
A security context defines privilege and access control settings for a Pod or Container. Security context settings include, but are not limited to:

Discretionary Access Control: Permission to access an object, like a file, is based on user ID (UID) and group ID (GID).

Security Enhanced Linux (SELinux): Objects are assigned security labels.

Running as privileged or unprivileged.

Linux Capabilities: Give a process some privileges, but not all the privileges of the root user.

AppArmor: Use program profiles to restrict the capabilities of individual programs.

Seccomp: Filter a process's system calls.

allowPrivilegeEscalation: Controls whether a process can gain more privileges than its parent process. This bool directly controls whether the no_new_privs flag gets set on the container process. allowPrivilegeEscalation is always true when the container:

is run as privileged, or has CAP_SYS_ADMIN
readOnlyRootFilesystem: Mounts the container's root filesystem as read-only.

The above bullets are not a complete set of security context settings -- please see SecurityContext for a comprehensive list.

In practice

(taken from Kubernetes docs - https://Kubernetes.io/docs/tasks/configure-pod-container/security-context/)

Below is an example of what a security context policy looks like:

apiVersion: v1

kind: Pod

metadata:
  name: security-context-demo
spec:
  securityContext:
    runAsUser: 1000
    runAsGroup: 3000
    fsGroup: 2000
  volumes:
  - name: sec-ctx-vol
    emptyDir: {}
  containers:
  - name: sec-ctx-demo
    image: busybox:1.28
    command: [ "sh", "-c", "sleep 1h" ]
    volumeMounts:
    - name: sec-ctx-vol
      mountPath: /data/demo
    securityContext:
      allowPrivilegeEscalation: false

In the configuration file, the runAsUser field specifies that for any containers in the Pod, all processes run with user ID 1000. The runAsGroup field specifies the primary group ID of 3000 for all processes within any containers of the Pod. If this field is omitted, the primary group ID of the containers will be root(0). Any files created will also be owned by user 1000 and group 3000 when runAsGroup is specified. Since fsGroup field is specified, all processes of the container are also part of the supplementary group ID 2000. The owner for volume /data/demo and any files created in that volume will be Group ID 2000.

Managing Security Context in bulk

Creating security context and applying it to a single cluster is relatively easy. What happens when we need to apply it on multiple clusters over multiple pods/containers? It just so happens that there are many tools that make this process a lot easier. Below is a solution we have that allows you to set policy and apply it to multiple Kubernetes clusters, pods and containers.

Handle Security by Policy

The first thing you will notice is that you can see the policy of your pods/containers by just clicking on one.

The second screen illustrates baseline policy that can be deployed based on standard guidelines. You can also duplicate the policy and/or create your own here.

The real power comes in the form of a policy advisor, illustrated below in which the application analyzes the pods/containers and provides recommendations on the correct policy.

Lastly, the solution has a central policy that allows you to define the standard policy that includes elements around security context API security and vulnerabilities. We will discuss vulnerabilities and API security in our upcoming articles.

If you are interested in securing your cloud native workloads then try out Panoptica at Panoptica.app. There is a free tier that is always free!

Additionally, check out our Open Source in this area:

OpenClarity (https://openclarity.io/) - the first open-source solution that forms a cloud native application protection platform. Security needs to shift left into your development pipeline. OpenClarity allows you to understand and manage vulnerabilities and security configuration of Kubernetes, Serverless and APIs.

KubeClarity – Understand your software bill of materials (SBOM) and the vulnerabilities within. Scan and understand results from different CI stages and detect vulnerabilities in different stages of deployment. Group scanned resources (images/directories) under defined applications to navigate the object tree dependencies (applications, resources, packages, vulnerabilities)
APIClarity – Analyzes the API endpoint to illustrate intended usage, deprecated API calls, and undocumented features. Additionally, executes trace analyzers fuzzers and BFLA detection to detect potential security threats such as Broken Object Level Authorizations (BOLA), Broken Function Level Authorization(BFLA), and other security issues.
FunctionClarity - a code integrity solution for serverless functions. It allows users to sign their serverless functions and verify their integrity prior to their execution in their cloud environments. FunctionClarity includes a CLI tool, complemented by a "verification" function deployed in the target cloud account. The solution is designed for CI/CD insertion, where the serverless function code/images can be signed and uploaded before the function is created in the cloud repository.

How did we get from legacy to microservices?

Michael Chenetz — Fri, 20 Jan 2023 20:22:31 +0000

Ever wonder why a microservices based architecture came to exist? I often find it useful to understand the evolution of how technology came to be. In doing so, there is great understanding.

If we go back to the before times then we will find that application architecture was relatively simple. Most applications used a 3 tier design that would consist of a client, server and database. You had the client that the user used to connect to the server and display information. You had the database that contained all of the information that needed to be stored. Finally, you had the server that contained the logic for both backend and frontend calls.

We created multiple environments in order to have a level of redundancy and keep some semblance of uptime. A load-balancer would front-end these servers and send traffic to the different environments.

This was an okay solution when applications were updated quarterly and there was a lot of time to plan and test. Times have changed and needs have shifted. We now live in world ruled by SaaS (Software As A Service). The rules of engagement have changed. Consumers of applications need the latest features ASAP in order to push out their new widget and track that widget and ascertain the full lifecycle of that widget. I digress. But, You get the point.

Application design and architecture has changed to. At one time all calls were internal to the server and very little communicated to outside services. Now we live in a API centric world in which we consume most services through them. Smart developers are also looking to get the job done the easiest and quickest so APIs allow that to happen.

Developers are now having an easier time implementing APIs because of standards like OpenAPI spec. Developers started to think about loosely coupling their apps and deconstructing them like a trendy entree. What used to be libraries that were called from a single monolithic app are now designed to be micro-services and are split out by function. This allows developers to work on a single component of an application and they can update that piece, by itself, without effecting other components.

We can now decide where services should run and how they should communicate now that the services are split out by function. Additionally services can be scaled out so that they can handle additional load and traffic. We can now introduce services slowly and quickly phase them out if they are functioning improperly.

Hopefully this article highlights some of the core aspects of the shifts that occurred that led to micro-services based architecture.

Now that you a little bit of the history here is an article with some great information on how to determine the resource limits for a new microservice.
https://dev.to/ciscoemerge/determine-resource-limits-for-a-new-microservice-471

Check out some of our open-source solutions:
https://eti.cisco.com/open-source

Gitea, DroneCI and Portainer

Michael Chenetz — Fri, 30 Sep 2022 21:05:57 +0000

Intro

You are tasked to create a build environment and you want something relatively easy and something that is repeatable.

On my journey to accomplish the above task i came across many great solutions. There are all in one solutions like Gitlab and Github but i wanted a solution that was open-source, free and easy.

Gitea

Gitea
Gitea is a open-source version of something like Github that is easy to deploy and makes a great on-prem repository. An added advantage is it has out of the box integration with Drone that is configurable to trigger web-hooks on selected events.

Drone

Drone
Drone is a open-source CI tool that will take simple commands in the form of yaml and execute on them as a pipeline. They have a concept of runners that do all the work. You install the runners you need on Docker, or Kubernetes or even on a host and they will listen for jobs.

Runners:

Docker
Kubernetes
Exec
SSH
VM

Portainer

Portainer
Portainer is a container management platform that allows you to manage containers across clouds and allows you to manage multiple container platforms (Kubernetes, Docker, Docker Swarm, and Nomad)

Solution:

I am using VSCode as my IDE in which i have created a little python webserver that will be encapsulated into a docker image. The Docker image will then be created, sent to a repo and then consumed by a Kubernetes deployment. The deployment will be triggered and deployed.

Here is what it will look like:

[VSCode] -> [Gitea] -> [Drone] -> [Portainer] -> [Deploy]

Gitea Install:

Kubernetes:

helm install gitea gitea-charts/gitea -n gitea --values ./gitea_values.yml

gitea-values.yml (Changes)

ingress:
  enabled: true
  # className: nginx
  className:
  annotations: {}
    # kubernetes.io/ingress.class: nginx
    # kubernetes.io/tls-acme: "true"
  hosts:
    - host: git.chenetz.net
      paths:
        - path: /
          pathType: Prefix
  tls:
    - secretName: git-chenetz-tls
      hosts:
        - git.chenetz.net

Drone Install:

Kubernetes:

helm upgrade my-drone drone/drone -n drone --values ./drone-values.yaml

drone-values.yaml (changes)

ingress:
  enabled: true
  annotations: {}
    # kubernetes.io/ingress.class: nginx
    # kubernetes.io/tls-acme: "true"
  hosts:
    - host: drone.chenetz.net
      paths:
        - path: /
          pathType: Prefix
  tls: 
   - secretName: chenetz-drone-tls
     hosts:
       - drone.chenetz.net
nv:
  DRONE_WEBHOOK_SKIP_VERIFY: true
  ## REQUIRED: Set the user-visible Drone hostname, sans protocol.
  ## Ref: https://docs.drone.io/installation/reference/drone-server-host/
  ##
  DRONE_SERVER_HOST: drone.chenetz.net
  ## The protocol to pair with the value in DRONE_SERVER_HOST (http or https).
  ## Ref: https://docs.drone.io/installation/reference/drone-server-proto/
  ##
  DRONE_SERVER_PROTO: https

  ## REQUIRED: Set the secret secret token that the Drone server and its Runners will use
  ## to authenticate. This is commented out in order to leave you the ability to set the
  ## key via a separately provisioned secret (see existingSecretName above).
  ## Ref: https://docs.drone.io/installation/reference/drone-rpc-secret/
  ##
  DRONE_RPC_SECRET: drone-runner

  ## If you'd like to use a DB other than SQLite (the default), set a driver + DSN here.
  ## Ref: https://docs.drone.io/installation/storage/database/
  ##
  # DRONE_DATABASE_DRIVER:
  # DRONE_DATABASE_DATASOURCE:

  ## If you are going to store build secrets in the Drone database, it is suggested that
  ## you set a database encryption secret. This must be set before any secrets are stored
  ## in the database.
  ## Ref: https://docs.drone.io/installation/storage/encryption/
  ##
  # DRONE_DATABASE_SECRET:

  ## If you are using self-hosted GitHub or GitLab, you'll need to set this to true.
  ## Ref: https://docs.drone.io/installation/reference/drone-git-always-auth/
  ##
  # DRONE_GIT_ALWAYS_AUTH: false

  ## ===================================================================================
  ##                         Provider Directives (select ONE)
  ## -----------------------------------------------------------------------------------
  ## Select one provider (and only one). Refer to the corresponding documentation link
  ## before filling the values in. Also note that you can use the 'secretMounts' value
  ## if you'd rather not have secrets in Kubernetes Secret instead of a ConfigMap.
  ## ===================================================================================

  ## GitHub-specific variables. See the provider docs here:
  ## Ref: https://docs.drone.io/installation/providers/github/
  ##
  # DRONE_GITHUB_CLIENT_ID:
  # DRONE_GITHUB_CLIENT_SECRET:

  ## GitLab-specific variables. See the provider docs here:
  ## Ref: https://docs.drone.io/installation/providers/gitlab/
  ##
  # DRONE_GITLAB_CLIENT_ID:
  # DRONE_GITLAB_CLIENT_SECRET:
  # DRONE_GITLAB_SERVER:

  ## Bitbucket Cloud-specific variables. See the provider docs here:
  ## Ref: https://docs.drone.io/installation/providers/bitbucket-cloud/
  ##
  # DRONE_BITBUCKET_CLIENT_ID:
  # DRONE_BITBUCKET_CLIENT_SECRET:

  ## Bitbucket-specific variables. See the provider docs here:
  ## Ref: https://docs.drone.io/installation/providers/bitbucket-server/
  ##
  # DRONE_GIT_USERNAME:
  # DRONE_GIT_PASSWORD:
  # DRONE_STASH_CONSUMER_KEY:
  # DRONE_STASH_PRIVATE_KEY:
  # DRONE_STASH_SERVER:

  ## Gitea-specific variables. See the provider docs here:
  ## Ref: https://docs.drone.io/installation/providers/gitea/
  ##
  DRONE_GITEA_CLIENT_ID: [client id]
  DRONE_GITEA_CLIENT_SECRET: [secret]
  DRONE_GITEA_SERVER: https://git.chenetz.net

Drone Docker Runner Install:

Kubernetes:

helm install my-drone-runner-docker drone/drone-runner-docker -n drone --values ./drone-runner-docker.yml

drone-runner-docker.yml (changes)

env:
  ## The docker host to be used by the drone-runner-docker when connecting
  ## to the docker daemon, defaults to the dind(sidecar) docker daemon
  DOCKER_HOST: "tcp://localhost:2375"
  ## REQUIRED: Set the secret secret token that the Docker runner will use
  ## to authenticate. This is commented out in order to leave you the ability to set the
  ## key via a separately provisioned secret (see extraSecretNamesForEnvFrom above).
  ## Ref: https://docs.drone.io/runner/docker/configuration/reference/drone-rpc-secret/
  ##
  DRONE_RPC_SECRET: drone-runner

  ## The hostname/IP (and optionally the port) for your Kubernetes runner. Defaults to the "drone"
  ## service that the drone server Chart creates by default.
  ## Ref: https://docs.drone.io/runner/docker/configuration/reference/drone-rpc-host/
  ##
  DRONE_RPC_HOST: drone.chenetz.net

  ## The protocol to use for communication with Drone server.
  ## Ref: https://docs.drone.io/runner/docker/configuration/reference/drone-rpc-proto/
  ##
  DRONE_RPC_PROTO: https
  DRONE_NAMESPACE_DEFAULT: drone

Configuration:

Gitea:

Drone

After getting everything up and communicating, i added the .values.yml file to my repo. The following drone pipeline file creates the docker image after it gets a trigger from either a push or pull in Gitea. The second step triggers a webhook that is configured in Portainer to trigger a deployment of the manifest.yml in the repo.

kind: pipeline
name: default
type: docker 
steps:
- name: docker  
  image: plugins/docker
  settings:
    username:
      from_secret: docker-username
    password:
      from_secret: docker-password
    repo: mchenetz/test
    tags:
    - latest
- name: send
  image: plugins/webhook
  settings:
    urls: https://192.168.10.12:9443/api/stacks/webhooks/
    content_type: application/json
    template: |
      {
        "owner": "{{ repo.owner }}",
        "repo": "{{ repo.name }}",
        "status": "{{ build.status }}",
      }
    skip_verify: true
    debug: true

Dockerfile

FROM python:latest
WORKDIR /src

COPY . .

CMD [ "python", "./test.py"]

Source (test.py)

from http.server import BaseHTTPRequestHandler, HTTPServer
import time

hostName = "localhost"
serverPort = 80

class MyServer(BaseHTTPRequestHandler):
    def do_GET(self):
        self.send_response(200)
        self.send_header("Content-type", "text/html")
        self.end_headers()
        self.wfile.write(bytes("<html><head><title>https://portainer.io</title></head>", "utf-8"))
        self.wfile.write(bytes("<p>Request: %s</p>" % self.path, "utf-8"))
        self.wfile.write(bytes("<body>", "utf-8"))
        self.wfile.write(bytes("<p>This is an example web server on k8s.</p>", "utf-8"))
        self.wfile.write(bytes("</body></html>", "utf-8"))

if __name__ == "__main__":        
    webServer = HTTPServer((hostName, serverPort), MyServer)
    print("Server started http://%s:%s" % (hostName, serverPort))

    try:
        webServer.serve_forever()
    except KeyboardInterrupt:
        pass

    webServer.server_close()
    print("Server stopped.")

Example of automated trigger on push:

Docker Repo:

Portainer Trigger:

Conclusion:

At the end of the day... It may sound like a few steps to setup but i now have a complete environment for my coding pipeline and any changes i make will automatically build the image and deploy. That may not always be what you want and the pipeline can be modified to insert stops for approval and to change to pull which is more common.

Managing multiple Kube clusters (the easy way)

Michael Chenetz — Fri, 16 Sep 2022 20:00:34 +0000

If you have ever tried to manage users on multiple Kubernetes clusters then you know the pain i am about to talk about.

The first thing you need to do after you bring up a cluster is define some privileges. You accomplish that by setting up roles and cluster roles.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  creationTimestamp: "2022-07-20T23:53:17Z"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: cluster-admin
  resourceVersion: "76"
  uid: 15d14062-0879-4232-9f42-51b79c0835f9
rules:
- apiGroups:
  - '*'
  resources:
  - '*'
  verbs:
  - '*'
- nonResourceURLs:
  - '*'
  verbs:
  - '*'

After you have defined the rules in a role (outside of the scope of this article) then you need to create a clusterrolebinding or rolebinding that associated a particular user with a role or clusterrole. If this sounds overly complicated it's because it probably is.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  creationTimestamp: "2022-07-20T23:53:17Z"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: cluster-admin
  resourceVersion: "138"
  uid: 5ef726c1-01df-4ecd-8951-909698ef5472
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:masters

Oh yeah, and if forgot... You need to do this for every cluster you bring up.

I am here to tell you there is an easier way!

Easy Button

Using Portainer BE you can setup your users in the GUI.

You can then assign the users to groups, called teams.

Then you can associate the roles to groups per environment

You can now assign these same teams and roles to each envirnment. Kubernetes gets the user association in the backgrounds and the cluster roles are pushed down to each cluster. Now the only thing to do is to grab a kubeconfig file that is associated with that user.

All the clusters that user is associated with will be part of the kubeconfig and the kubeconfig points to a proxy that manages all of the calls. If a user is taken out of a team in portainer then those changes are immediate across clusters.

You can se how much easier it is to manage multiple cluster access this way. For more of a demo then checkout the video attached to this post.

Can DevOps and ITOps Coexist?

Michael Chenetz — Tue, 16 Aug 2022 20:57:13 +0000

It's the age old conflict between ITOps and DevOps. IT works bottom up and handles all of the infra related tasks from the network to the VM. DevOps works top down and looks at things from the application down to the container. Somewhere in the middle they each have to co-exist.

Really, there should be no conflict. What we think of as conflict is just consuming things in a way that makes sense for the associated audience. If we say that something is very IT centric then what we are really saying is that we are looking for some type of management interface that gives us nice dashboards and allows us to manage a bunch of resources at a time. If we say something is DevOps centric then what we are saying is that we are looking for a high level of automation. Usually that automation exhibits in the form of CLI commands and APIs.

The question remains on whether there can be a single solution to tie things together. The answer is that there can be. The best tools allow people to consume their infra and apps in a way they are accustomed to. At the end of the day, there are a few elements that are necessary for both ITOps and DevOps. Some of these elements are RBAC (Role Based Access Control) and automation.

In this episode of Portainerd, I go through an example on how the two different teams can work together and how RBAC enables it.

Up and running with K0s and Portainer

Michael Chenetz — Wed, 10 Aug 2022 15:26:00 +0000

Prerequisites:

Physical server
Ubuntu Linux x 4
network

Intro:

In this tutorial I will explain how to get started with K0s and Portainer. K0s is a great orchestrator for installing Kubernetes and makes it so that Kubernetes can be installed the same way many times. Portainer is a platform that runs on and can interact with most container platforms and allows you to manage your apps and micro-services.

In this tutorial we will be using Portainer BE which is free for up to 5 nodes. The BE edition has some great functionality around RBAC, Edge, and Cloud provisioning of Kubernetes clusters.

This intro does not cover the installation of Ubuntu onto a server and assumes that has already been done.

Installing K0s:

The first thing that you need to know is that K0S is a distributed install. You use the k0sctl tool to connect over ssh to your hosts and it will provision everything for you.

To start, install the k0sctl binary on the system you want to orchestrate from.

on Mac:

brew install k0sproject/tap/k0sctl

On Windows:

choco install k0sctl

On Linux/Bash:

k0sctl completion > /etc/bash_completion.d/k0sctl

On Linux/Zsh:

k0sctl completion > /usr/local/share/zsh/site-functions/_k0sctl

generating and deploying an SSH key

on the host that you are orchestrating from, do the following to create the key

ssh-keygen -t ed25519

Just hit enter until everything all prompts are gone and the key is created

to deploy the key do the following. The user should be whatever user you created on the ubuntu host and the ip should be the ip of each ubuntu host:

ssh-copy-id -i ~/.ssh/id_ed25519 user@host

Configuring K0s:

K0s needs a yaml file that holds the configuration of your cluster. K0s will actually create this yaml file for you in order to start out with. To do this, run the following command in a new directory:

k0sctl init > k0sctl.yaml

Once you have the file then edit it with the editor of your choice. I tend to choose vscode. Below is an example of a 4 node cluster with one controller and three workers. Each host has an ip address, a user name, the path of the ssh key(on the orchestrating machine) to authenticate to the host on connection, and the role of that host. The nice part about this is that once this is configured you can run it over and over.

apiVersion: k0sctl.k0sproject.io/v1beta1
kind: Cluster
metadata:
  name: k0s-cluster
spec:
  hosts:
  - ssh:
      address: 192.168.10.1
      user: user
      port: 22
      keyPath: /Users/user/.ssh/id_ed25519
    role: controller
  - ssh:
      address: 192.168.10.2
      user: user
      port: 22
      keyPath: /Users/user/.ssh/id_ed25519
    role: worker
  - ssh:
      address: 192.168.10.3
      user: user
      port: 22
      keyPath: /Users/user/.ssh/id_ed25519
    role: worker
  - ssh:
      address: 192.168.10.4
      user: user
      port: 22
      keyPath: /Users/user/.ssh/id_ed25519
    role: worker
  k0s:
    version: 1.24.2+k0s.0
    dynamicConfig: false

Once you have everything that you need configured then run the following command to bring everything up:

Mac/Linux:

k0sctl apply --config ./k0sctl.yaml

Windows:

k0sctl apply --config k0sctl.yaml

You will get a bunch of output from that command that looks like the following:

⠀⣿⣿⡇⠀⠀⢀⣴⣾⣿⠟⠁⢸⣿⣿⣿⣿⣿⣿⣿⡿⠛⠁⠀⢸⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠀█████████ █████████ ███
⠀⣿⣿⡇⣠⣶⣿⡿⠋⠀⠀⠀⢸⣿⡇⠀⠀⠀⣠⠀⠀⢀⣠⡆⢸⣿⣿⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀███          ███    ███
⠀⣿⣿⣿⣿⣟⠋⠀⠀⠀⠀⠀⢸⣿⡇⠀⢰⣾⣿⠀⠀⣿⣿⡇⢸⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠀███          ███    ███
⠀⣿⣿⡏⠻⣿⣷⣤⡀⠀⠀⠀⠸⠛⠁⠀⠸⠋⠁⠀⠀⣿⣿⡇⠈⠉⠉⠉⠉⠉⠉⠉⠉⢹⣿⣿⠀███          ███    ███
⠀⣿⣿⡇⠀⠀⠙⢿⣿⣦⣀⠀⠀⠀⣠⣶⣶⣶⣶⣶⣶⣿⣿⡇⢰⣶⣶⣶⣶⣶⣶⣶⣶⣾⣿⣿⠀█████████    ███    ██████████

INFO k0sctl 0.0.0 Copyright 2021, Mirantis Inc.
INFO Anonymized telemetry will be sent to Mirantis.
INFO By continuing to use k0sctl you agree to these terms:
INFO https://k0sproject.io/licenses/eula
INFO ==> Running phase: Connect to hosts
INFO [ssh] 10.0.0.1:22: connected
INFO [ssh] 10.0.0.2:22: connected
INFO ==> Running phase: Detect host operating systems
INFO [ssh] 10.0.0.1:22: is running Ubuntu 20.10
INFO [ssh] 10.0.0.2:22: is running Ubuntu 20.10
INFO ==> Running phase: Prepare hosts
INFO [ssh] 10.0.0.1:22: installing kubectl
INFO ==> Running phase: Gather host facts
INFO [ssh] 10.0.0.1:22: discovered 10.12.18.133 as private address
INFO ==> Running phase: Validate hosts
INFO ==> Running phase: Gather k0s facts
INFO ==> Running phase: Download K0s on the hosts
INFO [ssh] 10.0.0.2:22: downloading k0s 0.11.0
INFO [ssh] 10.0.0.1:22: downloading k0s 0.11.0
INFO ==> Running phase: Configure K0s
WARN [ssh] 10.0.0.1:22: generating default configuration
INFO [ssh] 10.0.0.1:22: validating configuration
INFO [ssh] 10.0.0.1:22: configuration was changed
INFO ==> Running phase: Initialize K0s Cluster
INFO [ssh] 10.0.0.1:22: installing k0s controller
INFO [ssh] 10.0.0.1:22: waiting for the k0s service to start
INFO [ssh] 10.0.0.1:22: waiting for kubernetes api to respond
INFO ==> Running phase: Install workers
INFO [ssh] 10.0.0.1:22: generating token
INFO [ssh] 10.0.0.2:22: writing join token
INFO [ssh] 10.0.0.2:22: installing k0s worker
INFO [ssh] 10.0.0.2:22: starting service
INFO [ssh] 10.0.0.2:22: waiting for node to become ready
INFO ==> Running phase: Disconnect from hosts
INFO ==> Finished in 2m2s
INFO k0s cluster version 0.11.0 is now installed
INFO Tip: To access the cluster you can now fetch the admin kubeconfig using:
INFO      k0sctl kubeconfig

The final output lets you know to run the command k0sctl kubeconfig which will generate the kubeconfig file that is needed to run kubectl (Kubernetes) commands.

To get the kubeconfig file run:

k0sctl kubeconfig > kubeconfig

To test the cluster run:

kubectl get pods --kubeconfig kubeconfig -A

You should see:

NAMESPACE     NAME                                       READY   STATUS    RESTARTS   AGE
kube-system   calico-kube-controllers-5f6546844f-w8x27   1/1     Running   0          3m50s
kube-system   calico-node-vd7lx                          1/1     Running   0          3m44s
kube-system   coredns-5c98d7d4d8-tmrwv                   1/1     Running   0          4m10s
kube-system   konnectivity-agent-d9xv2                   1/1     Running   0          3m31s
kube-system   kube-proxy-xp9r9                           1/1     Running   0          4m4s
kube-system   metrics-server-6fbcd86f7b-5frtn            1/1     Running   0          3m51s

K0s Issue:
There is an issue in some releases of K0s where certain directories were not in the expected locations for Kubernetes. Before you bang your head on the wall here is a fix. SSH to your worker nodes and run the following:

sudo ln -s /var/lib/k0s/kubelet /var/lib/

This gives you your base linux install but you still need a couple of other things. The first things is a load balancer. I typically use MetalLB. To install MetalLB run the following:

kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.10.2/manifests/namespace.yaml
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.10.2/manifests/metallb.yaml

To configure metalLB then edit the following with your network info. The network should correspond to free addresses on your subnet that you want to allocate when exposing containers externally through the load-balancer. Place the contents into your editor and save it as, "metallb.yaml"

metallb.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 192.168.10.240-192.168.10.250

To apply the configuration run:

kubectl apply -f ./metallb.yaml

The last thing i like to add is an ingress controller. I typically use Nginx for this. An ingress controller matches URL patterns to kubernetes services in order to save on using IP addresses for everything. You can create a single load balancer and then map DNS to that ip. The ingress will look at the referring DNS and match it. If there is a match it will follow the rules defined to connect the DNS match with a service.

To add ingress do the following:

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.3/deploy/static/provider/baremetal/deploy.yaml

You can have multiple ingress classes in your cluster. It's good to define a default. In order to do that run:

kubectl -n ingress-nginx annotate ingressclasses nginx ingressclass.kubernetes.io/is-default-class="true"

Lastly you need to change the default behavior of Nginx that would expose a nodeport to change to a LoadBalancer.

kubectl edit service ingress-nginx-controller -n ingress-nginx

Storage

This is a little outside the scope of the install but I will add it here because it could be useful to some.

A storage class is needed in Kubernetes in order to have data persistence. There are many ways to accomplish this. Some like to use a Kubernetes cluster as a storage class by using something like longhorn.io or Ceph. I have a NFS storage device that i setup on TrueNAS that i use for this purpose. The following instrutcions will setup a CSI (Container Storage Interface) with NFS:

Install NFS on the cluster:

helm repo add csi-driver-nfs https://raw.githubusercontent.com/kubernetes-csi/csi-driver-nfs/master/charts
helm install csi-driver-nfs csi-driver-nfs/csi-driver-nfs --namespace kube-system --version v4.1.0

apply storage class configuration. The only things that really need to be changed are the IP address of the NFS server and the share on that server:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nfs-csi
provisioner: nfs.csi.k8s.io
parameters:
  server: 192.168.10.10
  share: /mnt/default/code/
  # csi.storage.k8s.io/provisioner-secret is only needed for providing mountOptions in DeleteVolume
  # csi.storage.k8s.io/provisioner-secret-name: "mount-options"
  # csi.storage.k8s.io/provisioner-secret-namespace: "default"
reclaimPolicy: Delete
volumeBindingMode: Immediate
mountOptions:
  - nconnect=8  # only supported on linux kernel version >= 5.3
  - nfsvers=4.1

Lastly, we want to make that storage class default for the cluster:

kubectl patch storageclass nfs-csi -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

Notes:

One of the really nice features of K0s is automation. I haven't done it yet but you can automate the above extra steps by using the manifest deployer and helm deployer. Information is available below:

Manifest Deployer:
https://docs.k0sproject.io/v1.24.3+k0s.0/manifests/

Helm Deployer:
https://docs.k0sproject.io/v1.24.3+k0s.0/helm-charts/

Portainer Install:

Requirements:

Kubernetes cluster
Data persistent storage

Install:

helm repo add portainer https://portainer.github.io/k8s/
helm repo update

Installing via load-balancer is the easiest:

helm install --create-namespace -n portainer portainer portainer/portainer \
    --set service.type=LoadBalancer \
    --set enterpriseEdition.enabled=true \
    --set tls.force=true

If you want to get a little more advanced and have setup ingress and DNS and know the ingress load-balancer ip then run the following. Realize that there are many pre-requisites to get this working. Here is some info on Ingress (https://kubernetes.io/docs/concepts/services-networking/ingress/):

helm install --create-namespace -n portainer portainer portainer/portainer \
    --set enterpriseEdition.enabled=true \
    --set service.type=ClusterIP \
    --set tls.force=true \
    --set ingress.enabled=true \
    --set ingress.ingressClassName=<ingressClassName (eg: nginx)> \
    --set ingress.annotations."nginx\.ingress\.kubernetes\.io/backend-protocol"=HTTPS \
    --set ingress.hosts[0].host=<fqdn (eg: portainer.example.io)> \
    --set ingress.hosts[0].paths[0].path="/"

Logging in:

Load Balancer installed:

https://<loadbalancer IP>:9443/ or http://<loadbalancer IP>:9000/

Ingress installed:

https://<FQDN>/

Creating the first user:

Add your free license key:
If you have installed Portainer Business Edition, you will now be asked to provide your license key. You will have been provided this when signing up for Business Edition or the free trial. If you don't have a license key, you can either click the Don't have a license? link or get in touch with our team.
Paste the license key you were provided into the box and click Submit.

Import Kubernetes cluster:

Select the Kubernetes option and click Start Wizard. Then select the Import option.

Enter a name for cluster then click Select a file to browse for your kubeconfig file.

When you're ready, click the Connect button. If you have other environments to configure click Next to proceed, otherwise click Close to return to the list of environments where you will see the progress of your provision.

From here you can follow the docs to get everything else setup:
Portainer Docs

Check our more things you can do with portainer at (https://www.portainer.io/blog)

Get your 5 nodes free license at (https://www.portainer.io/pricing/take5)