Forem: Jonatan Ezron

Build own Kubernetes - Mid Sum up

Jonatan Ezron — Mon, 12 Dec 2022 20:58:13 +0000

I have been working on a new 'version' where the code is less coupled and more like Kubernetes where everything is a service, you can find the source code in here: https://github.com/jonatan5524/own-kubernetes

An updated article on the above can be found in Medium: https://medium.com/@jonatan5524/a-journey-of-writing-my-own-kubernetes-ef45839a769d

In this post, I will go into what we implemented so far from an architectural design perspective, how Kubernetes actually works from the inside and what will do going forward.

What we have done so far?

So Until this point, we implemented a basic creation of pods, nodes, services, and inner and outer networks of the nodes.
The flow is as followed:

So the client with the own-kubernetes CLI creates a node, and the node is created with Docker. Then the client sends an HTTP request to the IP and port of the node to create a pod and with that all of its networking.

How Kubernetes works from the inside?

So in Kubernetes, the design is that the master node (in our case our environment) has an etcd DB which is a key-value DB that is the source of truth of the cluster, and every interaction with the etcd is with the Kubernetes API.
The following flow is taken from the book Kubernetes in Action by Marko Luksa, highly recommended for anyone that wants to learn everything about Kubernetes and its inside:

So the way Kubernetes works is as the chart above, let's say we want to create a single Pod:
the kubectl makes a request to the Kubernetes API (usually some kubectl apply)

the API server saves the new pod spec in etcd
the scheduler gets a notification for a new Pod addition to the etcd without node assign and assigns a new node (with some decision conditions I will not get into)
the kubelet inside the node it was assigned to is getting a notification and creating the new pod.

There is also a another component that called controller manager, used to get the actual state of a resource to the desired state, usely update and delete resource and get also notification for every etcd change.

What we will do next?

So Kubernetes design is parted into components that avoid coupling, whereas our design is highly coupled and moving forward will probably be a monolith.
Our design will stay that way, we store the created resources in the etcd, but we will not work as decoupled components that are getting notifications, because I want this series to focus on getting a POC of minimal functionality of a working Kubernetes.
It was important to me to make this article before we move forward for you to realize the design we are doing is not a match to the Kubernetes design (but can be with extra work).

Build own Kubernetes - ClusterIP and NodePort in code

Jonatan Ezron — Thu, 17 Nov 2022 20:40:12 +0000

An updated article on the above can be found in Medium: https://medium.com/@jonatan5524/a-journey-of-writing-my-own-kubernetes-ef45839a769d

In the previous articles, we created the pod's network in code, in this article we are going to create a ClusterIP service.

To establish a ClusterIP service we needed to create some iptables rules, so first, we create some iptables handlers.
In pkg/net/iptables.go we create three functions: for creating a new chain, appending a new rule to the chain, and inserting in a specific location a rule.

package net

import (
    "fmt"

    "github.com/jonatan5524/own-kubernetes/pkg"
)

const (
    NAT_TABLE = "nat"
)

func NewIPTablesChain(name string) error {
    return pkg.ExecuteCommand(fmt.Sprintf("/usr/sbin/iptables -t %s -N %s", NAT_TABLE, name), true)
}

func AppendNewIPTablesRule(rule string, chain string) error {
    return pkg.ExecuteCommand(fmt.Sprintf("/usr/sbin/iptables -t %s -A %s %s", NAT_TABLE, chain, rule), true)
}

func InsertNewIPTablesRule(rule string, chain string, index int) error {
    return pkg.ExecuteCommand(fmt.Sprintf("/usr/sbin/iptables -t %s -I %s %d %s", NAT_TABLE, chain, index, rule), true)
}

The code above is fairly simple, we perform the commands we use in the previous article here.

Now we create a new file for service handling: pkg/service/service.go, in the file we first create some constraints, and a new service struct:

const (
    KUBE_SERVICES_CHAIN       = "KUBE-SERVICES"
    CLUSTER_IP_SERVICE_PREFIX = "KUBE-SVC"
    KUBE_SERVICE_MARK         = "KUBE-MARK-MASQ"
)

type ServiceType int

const (
    CluserIP ServiceType = iota
    NodePort
)

type Service struct {
    Type   ServiceType
    Id     string
    IpAddr string
}

The Service struct has 3 members:

Type - can be ClusterIP or NodePort
Id - unique generated ID
IpAddr - IP address of the service

Next, we create a new function for initializing the new iptables chains when the node is created:

func InitKubeServicesChain() error {
    // iptables -t nat -N KUBE-SERVICES
    if err := net.NewIPTablesChain(KUBE_SERVICES_CHAIN); err != nil {
        return err
    }

    // iptables -t nat -A PREROUTING -j KUBE-SERVICES
    if err := net.AppendNewIPTablesRule(fmt.Sprintf("-j %s", KUBE_SERVICES_CHAIN), "PREROUTING"); err != nil {
        return err
    }

    // iptables -t nat -A OUTPUT -j KUBE-SERVICES
    if err := net.AppendNewIPTablesRule(fmt.Sprintf("-j %s", KUBE_SERVICES_CHAIN), "OUTPUT"); err != nil {
        return err
    }

    if err := initMarkChain(); err != nil {
        return err
    }

    // iptables -t nat -A POSTROUTING -m mark --mark 0x4000/0x4000 -j MASQUERADE
    if err := net.AppendNewIPTablesRule("-m mark --mark 0x4000/0x4000 -j MASQUERADE", "POSTROUTING"); err != nil {
        return err
    }

    return nil
}

func initMarkChain() error {
    // iptables -t nat -N KUBE-MARK-MASQ
    if err := net.NewIPTablesChain(KUBE_SERVICE_MARK); err != nil {
        return err
    }

    // iptables -t nat -A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
    if err := net.AppendNewIPTablesRule("-j MARK --set-xmark 0x4000/0x4000", KUBE_SERVICE_MARK); err != nil {
        return err
    }

    return nil
}

As you can see above, the function creates a new KUBE-SERVICES chain and adds it to the PREROUTING and OUTPUT chain, adds new KUBE-MARK-MASQ chai and adds it to the POSTROUTING chain.
We call this function in pkg/agent/agent.go:

func setupIPTablesServices() {
    if err := service.InitKubeServicesChain(); err != nil {
        panic(err)
    }
}

func main() {
    startContainerd()
    setupIPTablesServices()

    e := echo.New()

    initMiddlewares(e)
    initRoutes(e)

    e.Logger.Fatal(e.Start(fmt.Sprintf(":%s", api.PORT)))
}

Next, we create a function that creates a new ClusterIP service:

func NewClusterIPService(port string, podsCidr string) (*Service, error) {
    const MAX_CHAIN_NAME = 29
    serviceName := pkg.GenerateNewID(CLUSTER_IP_SERVICE_PREFIX)[:MAX_CHAIN_NAME-len(CLUSTER_IP_SERVICE_PREFIX)-1]

    if err := net.NewIPTablesChain(serviceName); err != nil {
        return nil, err
    }

    ipAddr := "172.17.10.10"

    // iptables -t nat -I KUBE-SERVICES 1 ! -s podsCidr -d 1ipAddr -p tcp -m tcp --dport port -j KUBE-MARK-MASQ
    if err := net.InsertNewIPTablesRule(
        fmt.Sprintf("! -s %s -d %s -p tcp -m tcp --dport %s -j %s", podsCidr, ipAddr, port, KUBE_SERVICE_MARK),
        KUBE_SERVICES_CHAIN, 1); err != nil {
        return nil, err
    }

    // iptables -t nat -A KUBE-SERVICES -d clusterIP -p tcp -m tcp --dport port -j ServicerName
    if err := net.AppendNewIPTablesRule(
        fmt.Sprintf("-d %s -p tcp -m tcp --dport %s -j %s", ipAddr, port, serviceName),
        KUBE_SERVICES_CHAIN); err != nil {
        return nil, err
    }

    return &Service{
        Type:   CluserIP,
        Id:     serviceName,
        IpAddr: ipAddr,
    }, nil
}

The function does as follows:

generate a new service name
creates a new chain
currently, there is a hardcoded IP (in the future we will save the IPs in an etcd DB)
insert new rules for the KUBE-SERVICES chain to mask the packets that are directed for the clusterIP and port and direct them to the next chain.

Next, we have a new function that adds a new pod to the service:

func AddRouteToClusterIPService(ip string, port string, service string, index int) error {
    podService := fmt.Sprintf(service[:len(service)-3]+"-%d", index)

    if err := net.NewIPTablesChain(podService); err != nil {
        return err
    }

    // iptables -t nat -A podService -s podIp -j KUBE-MARK-MASQ
    if err := net.AppendNewIPTablesRule(fmt.Sprintf("-s %s -j %s", ip, KUBE_SERVICE_MARK), podService); err != nil {
        return err
    }

    // iptables -t nat -A podService -p tcp -m tcp -j DNAT --to-destination route
    if err := net.AppendNewIPTablesRule(fmt.Sprintf("-p tcp -m tcp -j DNAT --to-destination %s", fmt.Sprintf("%s:%s", ip, port)), podService); err != nil {
        return err
    }

    if index == 0 {
        // iptables -t nat -A serviceName -j podService
        return net.AppendNewIPTablesRule(fmt.Sprintf("-j %s", podService), service)
    }

    // iptables -t nat -A service -m statistic --mode nth --every index --packet 0 -j podService
    return net.InsertNewIPTablesRule(fmt.Sprintf("-m statistic --mode nth --every %d --packet 0 -j %s", index+1, podService), service, 1)
}

This function creates a new chain for the pod route, adds its route and mask, and adds the rule to the service chain with load balance.

To get the running pods' IPs for later, we add a new field for the RunningPod struct:

type RunningPod struct {
    IPAddr      string
    Pod         *Pod
    Task        *containerd.Task
    exitStatusC <-chan containerd.ExitStatus
}

And assign it when the pod is run:

func NewPodAndRun(imageRegistry string, name string) (*RunningPod, error) {
    pod, err := NewPod(imageRegistry, name)
    if err != nil {
        return nil, err
    }

    log.Printf("pod created: %s\n", pod.Id)
    log.Printf("starting pod\n")

    runningPod, err := pod.Run()
    if err != nil {
        return nil, err
    }

    log.Printf("setting up pod network\n")
    podIPAddr, err := connectToNetwork(pod.Id, (*runningPod.Task).Pid())
    if err != nil {
        return nil, err
    }

    runningPod.IPAddr = podIPAddr

    return runningPod, nil
}

Now, for testing we add in the CreatePod function in pkg/agent/api a creation of 3 pods and a ClusterIP service:

unc CreatePod(c echo.Context) error {
    podDto := new(podDTO)
    if err := c.Bind(podDto); err != nil {
        return err
    }

    runningPod, err := pod.NewPodAndRun(podDto.ImageRegistry, podDto.Name)
    if err != nil {
        return err
    }

    // ------- test ---
    log.Printf("creating cluster ip\n")
    clusterIPService, err := service.NewClusterIPService("3001", fmt.Sprintf("%s/24", runningPod.IPAddr))
    if err != nil {
        return err
    }

    if err := service.AddRouteToClusterIPService(runningPod.IPAddr, "8080", clusterIPService.Id, 0); err != nil {
        return err
    }

    runningPod2, err := pod.NewPodAndRun(podDto.ImageRegistry, podDto.Name)
    if err != nil {
        return err
    }
    if err := service.AddRouteToClusterIPService(runningPod2.IPAddr, "8080", clusterIPService.Id, 1); err != nil {
        return err
    }

    runningPod3, err := pod.NewPodAndRun(podDto.ImageRegistry, podDto.Name)
    if err != nil {
        return err
    }
    if err := service.AddRouteToClusterIPService(runningPod3.IPAddr, "8080", clusterIPService.Id, 2); err != nil {
        return err
    }
    // -------------

    return c.JSON(http.StatusCreated, podDTO{
        ImageRegistry: podDto.ImageRegistry,
        Name:          runningPod.Pod.Id,
    })
}

If we build, run and create a new pod:

# create a new pod (actually creates 3 in the test)
❯ curl -X POST localhost:49256/pods -H 'Content-Type: application/json' -d '{"name": "pod2", "image registry": "docker.io/cloudnativelabs/whats-my-ip:latest"}'
{"image registry":"docker.io/cloudnativelabs/whats-my-ip:latest","name":"pod2-10563f40-61ae-426b-b965-b468aeafe046"}
# now we call inside the node
❯ sudo docker exec -it fe4 /bin/bash
root@fe47726fb371:/agent# curl http://172.17.0.2:3001
HOSTNAME:fe47726fb371 IP:10.0.2.3
root@fe47726fb371:/agent# curl http://172.17.0.2:3001
HOSTNAME:fe47726fb371 IP:10.0.2.2
root@fe47726fb371:/agent# curl http://172.17.0.2:3001
HOSTNAME:fe47726fb371 IP:10.0.2.4
root@fe47726fb371:/agent# curl http://172.17.0.2:3001
HOSTNAME:fe47726fb371 IP:10.0.2.3
root@fe47726fb371:/agent# curl http://172.17.0.2:3001
HOSTNAME:fe47726fb371 IP:10.0.2.2
root@fe47726fb371:/agent# curl http://172.17.0.2:3001
HOSTNAME:fe47726fb371 IP:10.0.2.4
root@fe47726fb371:/agent# curl http://172.17.0.3:3001

And we got a ClusterIP generated in code!

Now the NodePort service is simple, first, let's move the NewClusterIPService and AddRouteToClusterIPService functions to a new file pkg/service/clusterIP for organizing.

Now we add a new file pkg/service/nodePort.go, and we create there a new function called NewNodePortService:

func NewNodePortService(port string, podsCidr string, nodeIP string) (*Service, *Service, error) {
    clusterIPService, err := NewClusterIPService("3001", podsCidr)
    if err != nil {
        return nil, nil, err
    }

    // iptables -t nat -N KUBE-NODEPORTS
    if err := net.NewIPTablesChain(NODE_PORT_SERVICE_CHAIN); err != nil {
        return nil, nil, err
    }

    // iptables -t nat -A KUBE-SERVICES -j KUBE-NODEPORTS
    if err := net.AppendNewIPTablesRule(fmt.Sprintf("-j %s", NODE_PORT_SERVICE_CHAIN), KUBE_SERVICES_CHAIN); err != nil {
        return nil, nil, err
    }

    // iptables -t nat -I KUBE-NODEPORTS 1 -p tcp -m tcp --dport port -j KUBE-MARK-MASQ
    if err := net.InsertNewIPTablesRule(fmt.Sprintf("-p tcp -m tcp --dport %s -j %s", port, KUBE_SERVICE_MARK), NODE_PORT_SERVICE_CHAIN, 1); err != nil {
        return nil, nil, err
    }

    // iptables -t nat -A KUBE-NODEPORTS -p tcp -m tcp --dport port -j clusterIPService
    if err := net.AppendNewIPTablesRule(fmt.Sprintf("-p tcp -m tcp --dport %s -j %s", port, clusterIPService.Id), NODE_PORT_SERVICE_CHAIN); err != nil {
        return nil, nil, err
    }

    return clusterIPService, &Service{
        Type:   NodePort,
        Id:     NODE_PORT_SERVICE_CHAIN,
        IpAddr: nodeIP,
    }, nil
}

When a new NodePort service is created, a new ClusterIP service is created as well, we add new iptables rules and chains, and returns from the function the new ClusterIP service and the new NodePort service.

We test the NodePort service in the same place we tested the ClusterIP service, pkg/agent/api.go:

func CreatePod(c echo.Context) error {
    podDto := new(podDTO)
    if err := c.Bind(podDto); err != nil {
        return err
    }

    runningPod, err := pod.NewPodAndRun(podDto.ImageRegistry, podDto.Name)
    if err != nil {
        return err
    }

    // ------- test ---
    log.Printf("creating cluster ip\n")
    localIPAddr, err := net.GetLocalIPAddr("eth0")
    if err != nil {
        return err
    }
    clusterIPService, _, err := service.NewNodePortService("30001", fmt.Sprintf("%s/24", runningPod.IPAddr), localIPAddr)
    if err != nil {
        return err
    }

    if err := service.AddRouteToClusterIPService(runningPod.IPAddr, "8080", clusterIPService.Id, 0); err != nil {
        return err
    }

    runningPod2, err := pod.NewPodAndRun(podDto.ImageRegistry, podDto.Name)
    if err != nil {
        return err
    }
    if err := service.AddRouteToClusterIPService(runningPod2.IPAddr, "8080", clusterIPService.Id, 1); err != nil {
        return err
    }

    runningPod3, err := pod.NewPodAndRun(podDto.ImageRegistry, podDto.Name)
    if err != nil {
        return err
    }
    if err := service.AddRouteToClusterIPService(runningPod3.IPAddr, "8080", clusterIPService.Id, 2); err != nil {
        return err
    }
    // -------------

    return c.JSON(http.StatusCreated, podDTO{
        ImageRegistry: podDto.ImageRegistry,
        Name:          runningPod.Pod.Id,
    })
}

And outside of the node:

❯ curl 172.18.0.2:30001
HOSTNAME:15c79bf78092 IP:10.0.2.3
❯ curl 172.18.0.2:30001
HOSTNAME:15c79bf78092 IP:10.0.2.2
❯ curl 172.18.0.2:30001
HOSTNAME:15c79bf78092 IP:10.0.2.4
❯ curl 172.18.0.2:30001
HOSTNAME:15c79bf78092 IP:10.0.2.3
❯ curl 172.18.0.2:30001
HOSTNAME:15c79bf78092 IP:10.0.2.2

And we got a NodePort generated in code! (don't forget to delete the test)

As always, the source code can be found here, and the changes on the ClusterIP can be found in this commit, and for the NodePort can be found in this commit.

Build own Kubernetes - Pods network in code

Jonatan Ezron — Sat, 05 Nov 2022 11:44:42 +0000

An updated article on the above can be found in Medium: https://medium.com/@jonatan5524/a-journey-of-writing-my-own-kubernetes-ef45839a769d

In the previous articles, we created the pod's network using network devices and iptables rules, in this article and the next we are going to establish this in code.
Keep in mind, we are going to add the network functionality to the kube-agent for an easier approach, in Kubernetes the component responsible for this part is the kube-proxy

First, we want to create a docker network when creating a node, so in pkg/node/node.go we create a new function:

func createNetwork(cli *client.Client, ctx context.Context) error {
    _, err := cli.NetworkInspect(ctx, NODE_DOCKER_NETWORK_NAME, types.NetworkInspectOptions{})
    if err == nil {
        return nil
    }

    newNetwork := types.NetworkCreate{IPAM: &network.IPAM{
        Driver: "default",
        Config: []network.IPAMConfig{{
            Subnet: NODE_CIDR,
        }},
    }}

    _, err = cli.NetworkCreate(context.Background(), NODE_DOCKER_NETWORK_NAME, newNetwork)
    if err != nil {
        return err
    }

    return nil
}

The above function creates a network using the provided name and node CIDR, both of the constraints define in pkg/node/constraints.go:

package node

import "github.com/jonatan5524/own-kubernetes/pkg/agent/api"

const (
    NODE_NAME                = "node"
    NODE_IMAGE               = "own-kube-node"
    NODE_PORT                = api.PORT + "/tcp"
    NODE_PORT_HOST_IP        = "0.0.0.0"
    MEMORY_LIMIT             = 2.9e+9 // 2900MB
    CPU_LIMIT                = 2
    NODE_HOST_MIN_PORT       = 10250
    NODE_HOST_MAX_PORT       = 10300
    NODE_DOCKER_NETWORK_NAME = "kube-net"
    NODE_CIDR                = "172.18.0.0/24"
)

And now we add the network creation in the NewNode function, at the start of the function and hostConfig and ContainerCreate:

func NewNode(cli *client.Client, ctx context.Context) (*Node, error) {
    err := createNetwork(cli, ctx)
    if err != nil {
        return nil, err
    }

.
.
.

    hostConfig := &container.HostConfig{
        PortBindings: nat.PortMap{
            nat.Port(fmt.Sprintf("%s/tcp", api.PORT)): []nat.PortBinding{
                {
                    HostIP:   NODE_PORT_HOST_IP,
                    HostPort: "0",
                },
            },
        },
        NetworkMode: NODE_DOCKER_NETWORK_NAME,
        Resources: container.Resources{
            Memory:    MEMORY_LIMIT,
            CPUShares: CPU_LIMIT,
        },
        Privileged: true,
    }

    _, err = cli.ContainerCreate(ctx, config, hostConfig, &network.NetworkingConfig{}, nil, id)
    if err != nil {
        return nil, err
    }

    return &Node{Id: id}, nil
}

Now we create in pkg/util.go an ExecuteCommand function for all the commands we are going to execute in the node:

func ExecuteCommand(command string) error {
    cmd := exec.Command("bash", "-c", command)
    cmd.Stdout = os.Stdout

    if err := cmd.Start(); err != nil {
        return err
    }

    log.Printf("%s logs: %s", command, cmd.Stdout)

    return nil
}

Now for the pod's network, we add in pkg/pod/service.go in NewPodAndRun a call for a new function to connect to the network:

func NewPodAndRun(imageRegistry string, name string) (string, error) {
    pod, err := NewPod(imageRegistry, name)
    if err != nil {
        return "", err
    }

    log.Printf("pod created: %s\n", pod.Id)
    log.Printf("starting pod\n")

    runningPod, err := pod.Run()
    if err != nil {
        return "", err
    }

    log.Printf("setting up pod network\n")
    if err := connectToNetwork(pod.Id, (*runningPod.Task).Pid()); err != nil {
        return "", err
    }

    return pod.Id, nil
}

The connectToNetwork function creates a bridge and VXLAN if necessary, find a new available IP address and create the veth pair for connection:

func connectToNetwork(podId string, pid uint32) error {
    netId := podId[:15-len("veth")-1]

    podCIDR, err := generateNewNodePodCIDR()
    if err != nil {
        return err
    }

    // podCIDR: 10.0.2.0/24 -> bridgeIP: 10.0.2.1/24
    bridgeIP := pkg.ReplaceAtIndex(podCIDR, '1', len(podCIDR)-4)

    if !net.IsDeviceExists(BRIDGE_NAME) {
        if err := net.CreateBridge(BRIDGE_NAME, bridgeIP); err != nil {
            return err
        }
    }

    if !net.IsDeviceExists(VXLAN_NAME) {
        if err := net.CreateVXLAN(VXLAN_NAME, NODE_LOCAL_NETWORK_INTERFACE, BRIDGE_NAME); err != nil {
            return err
        }
    }

    podIP, err := net.GetNextAvailableIPAddr(podCIDR)
    if err != nil {
        return err
    }

    if err := net.CreateVethPairNamespaces(
        fmt.Sprintf("veth-%s", netId),
        fmt.Sprintf("ceth-%s", netId),
        BRIDGE_NAME,
        int(pid),
        podIP+podCIDR[len(podCIDR)-3:],
        bridgeIP,
    ); err != nil {
        return err
    }

    return nil
}

We generate a pod CIDR for the current pod using the generateNewNodePodCIDR function that gets the IP address of the node and switches the x in 10.0.x.0/24 the pod CIDR template with the last byte in the IP address. For example, if our node IP address is 172.18.0.2 the pod CIDR in this node will be 10.0.2.0/24.

func generateNewNodePodCIDR() (string, error) {
    localIPAddr, err := net.GetLocalIPAddr(NODE_LOCAL_NETWORK_INTERFACE)
    if err != nil {
        return "", err
    }

    // localIPAddr: 172.18.0.2 -> podCIDR: 10.0.2.0/24
    return strings.ReplaceAll(POD_CIDR, "x", string(localIPAddr[len(localIPAddr)-4])), nil
}

All the constraints are taken from pkg/pod/constraints.go:

package node

import "github.com/jonatan5524/own-kubernetes/pkg/agent/api"

const (
    NODE_NAME                = "node"
    NODE_IMAGE               = "own-kube-node"
    NODE_PORT                = api.PORT + "/tcp"
    NODE_PORT_HOST_IP        = "0.0.0.0"
    MEMORY_LIMIT             = 2.9e+9 // 2900MB
    CPU_LIMIT                = 2
    NODE_HOST_MIN_PORT       = 10250
    NODE_HOST_MAX_PORT       = 10300
    NODE_DOCKER_NETWORK_NAME = "kube-net"
    NODE_CIDR                = "172.18.0.0/24"
)

All of our network-related functions are from pkg/net/net.go, lets through those functions.
First, our create functions for veth, bridge, and vxlan is fairly simple, we just execute the commands from the previous articles:

func CreateBridge(name string, ipAddr string) error {
    if err := pkg.ExecuteCommand(fmt.Sprintf("ip link add %s type bridge", name)); err != nil {
        return err
    }

    if err := pkg.ExecuteCommand(fmt.Sprintf("ip addr add %s dev %s", ipAddr, name)); err != nil {
        return err
    }

    if err := pkg.ExecuteCommand(fmt.Sprintf("ip link set %s up", name)); err != nil {
        return err
    }

    return nil
}

func CreateVethPairNamespaces(name string, pair string, bridge string, namespacePID int, ipAddr string, bridgeIpAddr string) error {
    if err := pkg.ExecuteCommand(fmt.Sprintf("ip link add %s type veth peer name %s", name, pair)); err != nil {
        return err
    }

    if err := pkg.ExecuteCommand(fmt.Sprintf("ip link set %s up", name)); err != nil {
        return err
    }

    if err := pkg.ExecuteCommand(fmt.Sprintf("ip link set %s netns /proc/%d/ns/net", pair, namespacePID)); err != nil {
        return err
    }

    if err := pkg.ExecuteCommand(fmt.Sprintf("nsenter --net=/proc/%d/ns/net ip link set %s up", namespacePID, pair)); err != nil {
        return err
    }

    if err := pkg.ExecuteCommand(fmt.Sprintf("nsenter --net=/proc/%d/ns/net ip addr add %s dev %s", namespacePID, ipAddr, pair)); err != nil {
        return err
    }

    if err := pkg.ExecuteCommand(fmt.Sprintf("ip link set %s master %s", name, bridge)); err != nil {
        return err
    }

    if err := pkg.ExecuteCommand(fmt.Sprintf("nsenter --net=/proc/%d/ns/net /usr/sbin/ip route add default via %s", namespacePID, bridgeIpAddr)); err != nil {
        return err
    }

    return nil
}

func CreateVXLAN(name string, nodeInterface string, bridgeName string) error {
    const (
        ID    = "10"
        GROUP = "239.1.1.1"
    )

    if err := pkg.ExecuteCommand(fmt.Sprintf("ip link add %s type vxlan id %s group %s dstport 0 dev %s", name, ID, GROUP, nodeInterface)); err != nil {
        return err
    }

    if err := pkg.ExecuteCommand(fmt.Sprintf("ip link set %s master %s", name, bridgeName)); err != nil {
        return err
    }

    if err := pkg.ExecuteCommand(fmt.Sprintf("ip link set %s up", name)); err != nil {
        return err
    }

    return nil
}

Next, we have a function to check if a device exists, to check that, we can look for a file in /sys/class/net/ for example if a br0 device exists the /sys/class/net/br0 exists as well:

func IsDeviceExists(name string) bool {
    _, err := os.Stat(fmt.Sprintf("/sys/class/net/%s", name))

    return !os.IsNotExist(err)
}

Next, for getting the local IP address (used to discover the node IP address inside the node):

unc GetLocalIPAddr(interfaceName string) (addr string, err error) {
    ief, err := net.InterfaceByName(interfaceName)

    if err != nil {
        return "", err
    }

    addrs, err := ief.Addrs()
    if err != nil {
        return "", err
    }

    return addrs[0].String(), nil
}

And for getting a new available IP address with a given CIDR, we get all the possible hosts and ping each of them to see if one of them is not returning a response (meaning it is not allocated), this code was taken from ipcalc.go gist:

func GetNextAvailableIPAddr(cidr string) (string, error) {
    hosts, err := hosts(cidr)
    if err != nil {
        return "", err
    }

    for _, ip := range hosts {
        out, _ := exec.Command("ping", "-c1", "-t1", ip).Output()

        if strings.Contains(string(out), "Destination Host Unreachable") {
            return ip, nil
        }
    }

    return "", fmt.Errorf("no available ip have found in cidr: %s", cidr)
}

func hosts(cidr string) ([]string, error) {
    ip, ipnet, err := net.ParseCIDR(cidr)
    if err != nil {
        return nil, err
    }

    inc := func(ip net.IP) {
        for j := len(ip) - 1; j >= 0; j-- {
            ip[j]++
            if ip[j] > 0 {
                break
            }
        }
    }

    var ips []string
    for ip := ip.Mask(ipnet.Mask); ipnet.Contains(ip); inc(ip) {
        ips = append(ips, ip.String())
    }

    // remove network address and broadcast address
    return ips[1 : len(ips)-1], nil
}

Let's test those changes!

❯ make build
# create node
❯ sudo ./bin/main node create
2022/11/05 13:05:26 node created: node-9613851a-2fcb-440f-949d-77eade2c3b0b
2022/11/05 13:05:26 starting node
2022/11/05 13:05:26 node assign port: 49232
# create pod
❯ curl -X POST localhost:49232/pods -H 'Content-Type: application/json' -d '{"name": "pod1", "image registry": "docker.io/cloudnativelabs/whats-my-ip:latest"}'
{"image registry":"docker.io/cloudnativelabs/whats-my-ip:latest","name":"pod1-49fa480b-803c-4655-87a2-a93f643e0c61"}
# the node logs  - we can see all the commands that was executed
❯ sudo docker logs 892

   ____    __
  / __/___/ /  ___
 / _// __/ _ \/ _ \
/___/\__/_//_/\___/ v4.9.0
High performance, minimalist Go web framework
https://echo.labstack.com
____________________________________O/_______
                                    O\
2022/11/05 11:05:26 containerd logs: &{%!s(*os.file=&{{{0 0 0} 1 {0} <nil> 0 1 true true true} /dev/stdout <nil> false true false})}
2022/11/05 11:05:26 containerd running
⇨ http server started on [::]:10250
2022/11/05 11:05:37 pod created: pod1-49fa480b-803c-4655-87a2-a93f643e0c61
2022/11/05 11:05:37 starting pod
2022/11/05 11:05:37 setting up pod network
2022/11/05 11:05:37 ip link add br0 type bridge logs: &{%!s(*os.file=&{{{0 0 0} 1 {0} <nil> 0 1 true true true} /dev/stdout <nil> false true false})}
2022/11/05 11:05:37 ip addr add 10.0.3.1/24 dev br0 logs: &{%!s(*os.file=&{{{0 0 0} 1 {0} <nil> 0 1 true true true} /dev/stdout <nil> false true false})}
2022/11/05 11:05:37 ip link set br0 up logs: &{%!s(*os.file=&{{{0 0 0} 1 {0} <nil> 0 1 true true true} /dev/stdout <nil> false true false})}
2022/11/05 11:05:37 ip link add vxlan10 type vxlan id 10 group 239.1.1.1 dstport 0 dev eth0 logs: &{%!s(*os.file=&{{{0 0 0} 1 {0} <nil> 0 1 true true true} /dev/stdout <nil> false true false})}
2022/11/05 11:05:37 ip link set vxlan10 master br0 logs: &{%!s(*os.file=&{{{0 0 0} 1 {0} <nil> 0 1 true true true} /dev/stdout <nil> false true false})}
2022/11/05 11:05:37 ip link set vxlan10 up logs: &{%!s(*os.file=&{{{0 0 0} 1 {0} <nil> 0 1 true true true} /dev/stdout <nil> false true false})}
2022/11/05 11:05:50 ip link add veth-pod1-49fa4 type veth peer name ceth-pod1-49fa4 logs: &{%!s(*os.file=&{{{0 0 0} 1 {0} <nil> 0 1 true true true} /dev/stdout <nil> false true false})}
2022/11/05 11:05:50 ip link set veth-pod1-49fa4 up logs: &{%!s(*os.file=&{{{0 0 0} 1 {0} <nil> 0 1 true true true} /dev/stdout <nil> false true false})}
2022/11/05 11:05:50 ip link set ceth-pod1-49fa4 netns /proc/51/ns/net logs: &{%!s(*os.file=&{{{0 0 0} 1 {0} <nil> 0 1 true true true} /dev/stdout <nil> false true false})}
2022/11/05 11:05:50 nsenter --net=/proc/51/ns/net ip link set ceth-pod1-49fa4 up logs: &{%!s(*os.file=&{{{0 0 0} 1 {0} <nil> 0 1 true true true} /dev/stdout <nil> false true false})}
2022/11/05 11:05:50 nsenter --net=/proc/51/ns/net ip addr add 10.0.3.3/24 dev ceth-pod1-49fa4 logs: &{%!s(*os.file=&{{{0 0 0} 1 {0} <nil> 0 1 true true true} /dev/stdout <nil> false true false})}
2022/11/05 11:05:50 ip link set veth-pod1-49fa4 master br0 logs: &{%!s(*os.file=&{{{0 0 0} 1 {0} <nil> 0 1 true true true} /dev/stdout <nil> false true false})}
2022/11/05 11:05:50 nsenter --net=/proc/51/ns/net /usr/sbin/ip route add default via 10.0.3.1/24 logs: &{%!s(*os.file=&{{{0 0 0} 1 {0} <nil> 0 1 true true true} /dev/stdout <nil> false true false})}
method=POST, uri=/pods, status=201
# and inside the node we can see bridge and vxlan are created, and we can ping to the pod:
❯ sudo docker exec -it 892 /bin/bash
root@8923c146abc3:/agent# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
    link/ether 42:89:5f:a7:4f:78 brd ff:ff:ff:ff:ff:ff
    inet 10.0.3.1/24 scope global br0
       valid_lft forever preferred_lft forever
3: vxlan10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master br0 state UNKNOWN group default qlen 1000
    link/ether 42:89:5f:a7:4f:78 brd ff:ff:ff:ff:ff:ff
5: veth-pod1-49fa4@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UP group default qlen 1000
    link/ether b2:aa:10:8b:8a:25 brd ff:ff:ff:ff:ff:ff link-netnsid 1
267: eth0@if268: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 02:42:ac:12:00:03 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.18.0.3/24 brd 172.18.0.255 scope global eth0
       valid_lft forever preferred_lft forever
root@8923c146abc3:/agent# ping 10.0.3.3
PING 10.0.3.3 (10.0.3.3) 56(84) bytes of data.
64 bytes from 10.0.3.3: icmp_seq=1 ttl=64 time=0.142 ms
64 bytes from 10.0.3.3: icmp_seq=2 ttl=64 time=0.069 ms
^C
--- 10.0.3.3 ping statistics ---

In the next article, we will continue in our network implementation in code to ClusterIP and NodePort with iptables.

As always, the source code can be found here, the changes can be seen in this commit.

Build own Kubernetes - NodePort Service networking

Jonatan Ezron — Fri, 21 Oct 2022 11:58:14 +0000

An updated article on the above can be found in Medium: https://medium.com/@jonatan5524/a-journey-of-writing-my-own-kubernetes-ef45839a769d

In the previous articles, we established a connection between the pods across nodes using virtual network devices and some iptables rules.
In this article, we are going to make a NodePort service iptables rules to make a connection from outside of the cluster. For this article, I had the kubernetes-services-and-iptables blog helpful for the iptables rules in some services.
This article continues from the last article network architecture:

First, to make our rules organized and well structured we are going to make custom chains and add our rules there, we start by deleting all the rules from the last article and creating a new chain called KUBE-SERVICES and adding them to PREROUTING and OUTPUT chains so that every packet going through this chains will be affected by our rules:

iptables -t nat -N KUBE-SERVICES
iptables -t nat -A PREROUTING -j KUBE-SERVICES
iptables -t nat -A OUTPUT -j KUBE-SERVICES

Now we create a custom chain for our ClusterIP service called KUBE-SVC-1, and assign the desired destination IP (virtual cluster IP: 172.17.10.10) and port rule into the KUBE-SERVICE:

iptables -t nat -N KUBE-SVC-1
iptables -t nat -A KUBE-SERVICES -d 172.17.10.10/16 -p tcp -m tcp --dport 3001 -j KUBE-SVC-1

Now for each request to the ClusterIP service we want to load balance every request to the existing pod as before, so we add the DNAT statistics rules as before to the KUBE-SVC-1:

iptables -t nat -A KUBE-SVC-1 -m statistic --mode nth --every 4 --packet 0 -p tcp -m tcp -j DNAT --to-destination 10.0.1.2:8080
iptables -t nat -A KUBE-SVC-1 -m statistic --mode nth --every 3 --packet 0 -p tcp -m tcp -j DNAT --to-destination 10.0.1.3:8080
iptables -t nat -A KUBE-SVC-1 -m statistic --mode nth --every 2 --packet 0 -p tcp -m tcp -j DNAT --to-destination 10.0.2.2:8080
iptables -t nat -A KUBE-SVC-1 -p tcp -m tcp -j DNAT --to-destination 10.0.2.3:8080

If you list all the iptables rules the output should be as followed:

root@91cda5c7dade:/agent# iptables --list -n -t nat --line-number
Chain PREROUTING (policy ACCEPT)
num  target     prot opt source               destination
1    KUBE-SERVICES  all  --  0.0.0.0/0            0.0.0.0/0

Chain INPUT (policy ACCEPT)
num  target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
num  target     prot opt source               destination
1    KUBE-SERVICES  all  --  0.0.0.0/0            0.0.0.0/0

Chain POSTROUTING (policy ACCEPT)
num  target     prot opt source               destination

Chain KUBE-MARK-MASQ (0 references)
num  target     prot opt source               destination
1    MARK       all  --  0.0.0.0/0            0.0.0.0/0            MARK or 0x4000

Chain KUBE-NODEPORTS (1 references)
num  target     prot opt source               destination
1    KUBE-SVC-1  tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:30001

Chain KUBE-SERVICES (2 references)
num  target     prot opt source               destination
1    KUBE-SVC-1  tcp  --  0.0.0.0/0            172.17.0.0/16        tcp dpt:3001
2    KUBE-NODEPORTS  all  --  0.0.0.0/0            0.0.0.0/0

Chain KUBE-SVC-1 (2 references)
num  target     prot opt source               destination
1    DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            statistic mode nth every 4 tcp to:10.0.1.2:8080
2    DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            statistic mode nth every 3 tcp to:10.0.1.3:8080
3    DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            statistic mode nth every 2 tcp to:10.0.2.2:8080
4    DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp to:10.0.2.3:8080

If you make a request to http://172.17.10.10:3001 from inside the node or pods it will work as before.

Now for the NodePort it should be fairly easy, a NodePort service opens selected ports on all nodes between a port range 30000-32767 so that a client can request a node with this port to access a service.
Basically, we need to forward incoming requests from a specific port to our ClusterIP load balancer service that already exists.

Let's say we pick a NodePort port 3001, create a NodePort chain, and add them to our KUBE-SERVICES chain:

iptables -t nat -N KUBE-NODEPORTS
iptables -t nat -A KUBE-NODEPORTS -p tcp -m tcp --dport 30001 -j KUBE-SVC-1
iptables -t nat -A KUBE-SERVICES -j KUBE-NODEPORTS

If we make a request from our environment outside of the node to NodeIP:3001:

❯ curl http://172.17.0.3:30001
HOSTNAME:91cda5c7dade IP:10.0.1.2
❯ curl http://172.17.0.3:30001
HOSTNAME:91cda5c7dade IP:10.0.1.3
❯ curl http://172.17.0.3:30001
^C

The third request hangs and not returning a response, this is because the last 2 requests are for the second node, when the request responds to the process outside of the cluster (our host) it expects a different source IP, so we need to MASQUERADE every outgoing request, Kubernetes doing that by marking every service request with MARK module and then MASQUERADE the marked packets.
Let's start by adding a KUBE-MARK-MASQ chain, add a mark rule in it, and add it as the first rule for the KUBE-SERVICES with our ClusterIP rules, KUBE-NODEPORTS without node port rules:

iptables -t nat -N KUBE-MARK-MASQ
iptables -t nat -A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
iptables -t nat -I KUBE-SERVICES 1 ! -s 10.0.0.0/16 -d 172.17.10.10/16 -p tcp -m tcp --dport 3001 -j KUBE-MARK-MASQ
iptables -t nat -I KUBE-NODEPORTS 1 -p tcp -m tcp --dport 30001 -j KUBE-MARK-MASQ

Now we also want to mark each request forwarded to our pods, so we create a chain for each pod forward called KUBE-SVC-1-X, when X is just a serial number, mark the packet and forward to the relevant pod in each chain:

# pod1
iptables -t nat -N KUBE-SVC-1-1
iptables -t nat -A KUBE-SVC-1-1 -s 10.0.1.2/16 -j KUBE-MARK-MASQ
iptables -t nat -A KUBE-SVC-1-1 -p tcp -m tcp -j DNAT --to-destination 10.0.1.2:8080
# pod2
iptables -t nat -N KUBE-SVC-1-2
iptables -t nat -A KUBE-SVC-1-2 -s 10.0.1.3/16 -j KUBE-MARK-MASQ
iptables -t nat -A KUBE-SVC-1-2 -p tcp -m tcp -j DNAT --to-destination 10.0.1.3:8080
# pod3
iptables -t nat -N KUBE-SVC-1-3
iptables -t nat -A KUBE-SVC-1-3 -s 10.0.2.2/16 -j KUBE-MARK-MASQ
iptables -t nat -A KUBE-SVC-1-3 -p tcp -m tcp -j DNAT --to-destination 10.0.2.2:8080
# pod4
iptables -t nat -N KUBE-SVC-1-4
iptables -t nat -A KUBE-SVC-1-4 -s 10.0.2.3/16 -j KUBE-MARK-MASQ
iptables -t nat -A KUBE-SVC-1-4 -p tcp -m tcp -j DNAT --to-destination 10.0.2.3:8080

And add these chains to the KUBE-SVC-1 chain instead of the rules we wrote (delete the KUBE-SVC-1 rules before this command):

iptables -t nat -A KUBE-SVC-1 -m statistic --mode nth --every 4 --packet 0 -j KUBE-SVC-1-1
iptables -t nat -A KUBE-SVC-1 -m statistic --mode nth --every 3 --packet 0 -j KUBE-SVC-1-2
iptables -t nat -A KUBE-SVC-1 -m statistic --mode nth --every 2 --packet 0 -j KUBE-SVC-1-3
iptables -t nat -A KUBE-SVC-1 -j KUBE-SVC-1-4

Now we add a MASQUERADE rule for the POSTROUTING chain:

 iptables -t nat -A POSTROUTING -m mark --mark 0x4000/0x4000 -j MASQUERADE

Don't forget to add all the iptables rules we added to the second node as well!

We just need to change the last NodePort rule in KUBE-SERVICES to match every local package:

# delete the old one
iptables -t nat -D KUBE-SERVICES 3
iptables -t nat -A KUBE-SERVICES -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS

All of your iptables rules in the nat table should be as followed:

root@91cda5c7dade:/agent# iptables --list -n -t nat --line-number
Chain PREROUTING (policy ACCEPT)
num  target     prot opt source               destination
1    KUBE-SERVICES  all  --  0.0.0.0/0            0.0.0.0/0

Chain INPUT (policy ACCEPT)
num  target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
num  target     prot opt source               destination
1    KUBE-SERVICES  all  --  0.0.0.0/0            0.0.0.0/0

Chain POSTROUTING (policy ACCEPT)
num  target     prot opt source               destination
1    MASQUERADE  all  --  0.0.0.0/0            0.0.0.0/0            mark match 0x4000/0x4000

Chain KUBE-MARK-MASQ (6 references)
num  target     prot opt source               destination
1    MARK       all  --  0.0.0.0/0            0.0.0.0/0            MARK or 0x4000

Chain KUBE-NODEPORTS (1 references)
num  target     prot opt source               destination
1    KUBE-MARK-MASQ  tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:30001
2    KUBE-SVC-1  tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:30001

Chain KUBE-SERVICES (2 references)
num  target     prot opt source               destination
1    KUBE-MARK-MASQ  tcp  -- !10.0.0.0/16          172.17.0.0/16        tcp dpt:3001
2    KUBE-SVC-1  tcp  --  0.0.0.0/0            172.17.0.0/16        tcp dpt:3001
3    KUBE-NODEPORTS  all  --  0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL

Chain KUBE-SVC-1 (2 references)
num  target     prot opt source               destination
1    KUBE-SVC-1-1  all  --  0.0.0.0/0            0.0.0.0/0            statistic mode nth every 4
2    KUBE-SVC-1-2  all  --  0.0.0.0/0            0.0.0.0/0            statistic mode nth every 3
3    KUBE-SVC-1-3  all  --  0.0.0.0/0            0.0.0.0/0            statistic mode nth every 2
4    KUBE-SVC-1-4  all  --  0.0.0.0/0            0.0.0.0/0

Chain KUBE-SVC-1-1 (1 references)
num  target     prot opt source               destination
1    KUBE-MARK-MASQ  all  --  10.0.0.0/16          0.0.0.0/0
2    DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp to:10.0.1.2:8080

Chain KUBE-SVC-1-2 (1 references)
num  target     prot opt source               destination
1    KUBE-MARK-MASQ  all  --  10.0.0.0/16          0.0.0.0/0
2    DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp to:10.0.1.3:8080

Chain KUBE-SVC-1-3 (1 references)
num  target     prot opt source               destination
1    KUBE-MARK-MASQ  all  --  10.0.0.0/16          0.0.0.0/0
2    DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp to:10.0.2.2:8080

Chain KUBE-SVC-1-4 (1 references)
num  target     prot opt source               destination
1    KUBE-MARK-MASQ  all  --  10.0.0.0/16          0.0.0.0/0
2    DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp to:10.0.2.3:8080

And now if we make the request from our host:

❯ curl http://172.17.0.3:30001
HOSTNAME:91cda5c7dade IP:10.0.1.2
❯ curl http://172.17.0.3:30001
HOSTNAME:91cda5c7dade IP:10.0.1.3
❯ curl http://172.17.0.3:30001
HOSTNAME:f76b32aa3747 IP:10.0.2.2
❯ curl http://172.17.0.3:30001
HOSTNAME:f76b32aa3747 IP:10.0.2.3

Everything is working!

All of the work we have done in the last 2 posts need to happen automatically on each pod creation and deletion, we will automate this process in the next article in code!

Build own Kubernetes - ClusterIP Service networking

Jonatan Ezron — Thu, 20 Oct 2022 19:31:58 +0000

An updated article on the above can be found in Medium: https://medium.com/@jonatan5524/a-journey-of-writing-my-own-kubernetes-ef45839a769d

In the previous articles, we automated the node creation with Docker. Everything looked good, we saw we had a running node with a REST API agent and inside it a running pod, but what about the connection to the pod? the communication to the pod? if you try to send a request to the running pod in your environment it will not work, and even in the Node itself, only inside the running containerd to localhost, it will work.
So we need to solve this problem.
This part was very difficult for me, I needed to get deep into Linux namespaces and networking to solve it, this amazing blogs helped me a lot ontainer-networking-is-simple, some of the commands and drawing even was taken from there, I will explain the basics as we go along, but I recommend on reading this article before reading this, this article flow will be somewhat the same as iximiuz and msazure blogs.

All of the commands we will execute will be inside the Node container, so start by creating a node and entering into his shell:

❯ sudo ./bin/main node create
2022/10/14 16:00:16 node created: node-f8c7d6b8-ef24-4406-84e2-284e1f520bdf
2022/10/14 16:00:16 starting node
2022/10/14 16:00:16 node assign port: 49171
❯ sudo docker ps
CONTAINER ID   IMAGE           COMMAND     CREATED          STATUS          PORTS                      NAMES
73c64ab768c2   own-kube-node   "./agent"   28 seconds ago   Up 27 seconds   0.0.0.0:49171->10250/tcp   node-f8c7d6b8-ef24-4406-84e2-284e1f520bdf
❯ sudo docker exec -it 73c /bin/bash
root@73c64ab768c2:/agent#

Now we are going to use the ip, iptables and ping commands which are not installed in the container (later we will add them to the Dockerfile):

root@73c64ab768c2:/agent# apt-get install -y iproute2 iptables iputils-ping

Let's start and create 2 containers with ctr, we will use the whats-my-ip image so later it will be easy to test the connection.

root@73c64ab768c2:/agent# ctr image pull docker.io/cloudnativelabs/whats-my-ip:latest
root@73c64ab768c2:/agent# ctr run -d docker.io/cloudnativelabs/whats-my-ip:latest pod1
root@73c64ab768c2:/agent# ctr run -d docker.io/cloudnativelabs/whats-my-ip:latest pod2
root@73c64ab768c2:/agent# ctr c ls
CONTAINER    IMAGE                                           RUNTIME
pod1         docker.io/cloudnativelabs/whats-my-ip:latest    io.containerd.runc.v2
pod2         docker.io/cloudnativelabs/whats-my-ip:latest    io.containerd.runc.v2
root@73c64ab768c2:/agent# ctr t ls
TASK    PID     STATUS
pod2    974     RUNNING
pod1    920     RUNNING

We list the containers and tasks, as you can see there is no IP address assigned to the containers so we cant address them.
When a container is created for each instance a Linux namespace is created that isolates them from the host environment, our current state is looking like this:

The node has the eth0 interface for communication outside and the 2 containers have nothing.
So we need to create a virtual Ethernet device (veth), this is a virtual device that Linux provides us so we will able to communicate between namespace, these veth devices come in pairs:

root@73c64ab768c2:/agent# ip link add veth0 type veth peer name ceth0

Now we need one veth in our namespace and another in the container namespace, the container namespaces are located in /proc/PID/ns/net, so now we move ceth0 to the pod network namespace, activate the devices, and assign an IP address to ceth0:

# move ceth0 to pod1 network namespace
root@73c64ab768c2:/agent# ip link set ceth0 netns /proc/920/ns/net
# activate ceth0 in the test network namespace
root@73c64ab768c2:/agent# nsenter --net=/proc/920/ns/net ip link set ceth0 up
# assign IP address to ceth0
root@73c64ab768c2:/agent# nsenter --net=/proc/920/ns/net ip addr add 10.0.1.2/16 dev ceth0
# activate veth0 
root@73c64ab768c2:/agent# ip link set veth0 up

Now we will do the same for pod2 network and assign a different IP address:

root@73c64ab768c2:/agent# ip link add veth1 type veth peer name ceth1
root@73c64ab768c2:/agent# ip link set veth1 up
root@73c64ab768c2:/agent# ip link set ceth1 netns /proc/974/ns/net
root@73c64ab768c2:/agent# nsenter --net=/proc/974/ns/net ip link set ceth1 up
root@73c64ab768c2:/agent# nsenter --net=/proc/974/ns/net ip addr add 10.0.1.3/16 dev ceth1

Now our system looks like that:

Now we need to add a virtual bridge to separate the two networks we have created:

# creating a new bridge
root@73c64ab768c2:/agent# ip link add br0 type bridge
# activating the device
root@73c64ab768c2:/agent# ip link set br0 up
# connecting the two veth to the bridge
root@73c64ab768c2:/agent# ip link set veth0 master br0
root@73c64ab768c2:/agent# ip link set veth1 master br0
# assign an address to the bridge
root@73c64ab768c2:/agent# ip addr add 10.0.1.1/16 dev br0

our system will look like this:

We want also to make the bridge for each of the container's network its default gateway, we do this using the following commands:

root@73c64ab768c2:/agent# nsenter --net=/proc/920/ns/net ip route add default via 10.0.1.1
root@73c64ab768c2:/agent# nsenter --net=/proc/974/ns/net ip route add default via 10.0.1.1

Great we have two pods connected using a bridge and they both have two IP addresses.

Now we will focus on ClusterIP service, this service is for communication and load balancer inside the cluster between other pods across nodes.

Firstly we need to create a new node and two pods inside it and create the veth and bridge as we followed here, the IP is as followed:
pod3 10.0.2.2/16
pod4 10.0.2.3/16
bridge 10.0.2.1/16

# installing packages
apt-get install iproute2 iptables -y

# setup pod3 - 537 pid
ip link add veth0 type veth peer name ceth0
ip link set veth0 up
ip link set ceth0 netns /proc/537/ns/net
nsenter --net=/proc/537/ns/net ip link set ceth0 up
nsenter --net=/proc/537/ns/net ip addr add 10.0.2.2/16 dev ceth0

# setup pod2 - 595 pid
ip link add veth1 type veth peer name ceth1
ip link set veth1 up
ip link set ceth1 netns /proc/595/ns/net
nsenter --net=/proc/595/ns/net ip link set ceth1 up
nsenter --net=/proc/595/ns/net ip addr add 10.0.2.3/16 dev ceth1

# setup bridge
ip link add br0 type bridge
ip link set br0 up
ip addr add 10.0.2.1/16 dev br0
ip link set veth0 master br0
ip link set veth1 master br0

# setup default route for both pods
nsenter --net=/proc/537/ns/net ip route add default via 10.0.2.1
nsenter --net=/proc/595/ns/net ip route add default via 10.0.2.1

Our system is as followed:

Now we want to allow communication for the pods between nodes, we are doing this by using vxlan, which is an extended vlan method for virtualizing a LAN connection between different hosts.
we create that with the following command in the two nodes:

# create vxlan
ip link add vxlan10 type vxlan id 10 group 239.1.1.1 dstport 0 dev eth0
# connect vxlan to the bridge
ip link set vxlan10 master br0
# activate the vxlan
ip link set vxlan10 up

Our system is as followed:

Now for creating a ClusterIP service, we need to add iptables rules.
We will use the nat table and DNAT chain to address every request to the cluster IP (virtual selected IP) to a pod the command will look like this:

iptables -t nat -A PREROUTING -d $CLUSTER_IP -p tcp -m tcp --dport $CLUSTER_IP_PORT -j DNAT --to-destination $POD_IP4:$POD_PORT4

Now we want to include also load balancing using the round-robin method, iptables provide a solution for this too, using statistics, to forward each request by every option.
The full command looks like this:

iptables -t nat -A PREROUTING -d 172.17.10.10 -p tcp -m tcp --dport 3001 -m statistic --mode nth --every 4 --packet 0  -j DNAT --to-destination 10.0.1.2:8080
iptables -t nat -A PREROUTING -d 172.17.10.10 -p tcp -m tcp --dport 3001 -m statistic --mode nth --every 3 --packet 0  -j DNAT --to-destination 10.0.1.3:8080
iptables -t nat -A PREROUTING -d 172.17.10.10 -p tcp -m tcp --dport 3001 -m statistic --mode nth --every 2 --packet 0  -j DNAT --to-destination 10.0.2.2:8080
iptables -t nat -A PREROUTING -d 172.17.10.10 -p tcp -m tcp --dport 3001 -j DNAT --to-destination 10.0.2.3:8080

In the above commands, we establish four rules for every request to a virtual cluster IP load balance to our four pods using the round-robin method.

For testing the ClusterIP service we create a new pod in one of the nodes containing simple Linux like alpine, connect it to the network and bridge, and make a request, for each request you can see there is a different IP response!
In alpine curl is not installed by default and we don't have a communication to the outside world so we need to use wget:

wget http://172.17.10.10:3001
Connecting to 172.17.10.10:3001 (172.17.10.10:3001)
saving to 'index.html'
index.html           100% |********************************|    34  0:00:00 ETA
'index.html' saved
cat index.html
HOSTNAME:91cda5c7dade IP:10.0.1.2
rm index.html
wget http://172.17.10.10:3001
Connecting to 172.17.10.10:3001 (172.17.10.10:3001)
saving to 'index.html'
index.html           100% |********************************|    34  0:00:00 ETA
'index.html' saved
cat index.html
HOSTNAME:91cda5c7dade IP:10.0.1.3
rm index.html
wget http://172.17.10.10:3001
Connecting to 172.17.10.10:3001 (172.17.10.10:3001)
saving to 'index.html'
index.html           100% |********************************|    34  0:00:00 ETA
'index.html' saved
cat index.html
HOSTNAME:91cda5c7dade IP:10.0.2.2
rm index.html
wget http://172.17.10.10:3001
Connecting to 172.17.10.10:3001 (172.17.10.10:3001)
saving to 'index.html'
index.html           100% |********************************|    34  0:00:00 ETA
'index.html' saved
cat index.html
HOSTNAME:91cda5c7dade IP:10.0.2.3

We covered here a lot of Linux networking, I recommend if you wonder how I got to these commands and structure read the article I mentioned above. reference

All the networking we did above is done for every pod creation and deletion, for this, Kubernetes created some plugin for the containerd ctr called cni, I have decided to work on my own to learn better the underline of things.

We node have communication between pods but not communication to the outside world, in the next article, we will make a NodePort service!

Build own Kubernetes - Node commands

Jonatan Ezron — Wed, 12 Oct 2022 16:33:42 +0000

An updated article on the above can be found in Medium: https://medium.com/@jonatan5524/a-journey-of-writing-my-own-kubernetes-ef45839a769d

In the previous post, we built the node image and wrote the agent, all done by executing commands on the shell with docker build and run, but the whole process should be automated by our program, this is what this article will focus on, on creating node on command, listing them and deleting them.

Before we begin I had some minor fixes in the code: rename the command variables in cmd/pod.go because we are going to use the same name in node, change the agent pods routes to pods and not pod as BP in REST, and moved the GenerateNewID function to pkg/util.go. Some big change was in the Dockerfile, I am relying on the environment that is running the commands to have locally the node image built, so I made a Makefile to build the main program, the agent, and the node image.
Makefile:

default: build

build: build-agent build-node-image
    go build -o bin/main main.go

build-node-image: 
    sudo docker build -t own-kube-node .

build-agent: 
    go build -o bin/agent pkg/agent/agent.go

As a result, we can rely on the binary for the agent is already built when we build out the image so I removed the build from the Dockerfile:

# Dockerfile for node image
FROM ubuntu

WORKDIR /agent

RUN apt-get update && apt-get install -y wget containerd

COPY bin/agent .

EXPOSE 10250

ENTRYPOINT [ "./agent" ]

Let's start with our node functionality, Like the pod, we have related files to node functionality in pkg/node.
We start on pkg/node/constraints.go to define some constraints for our work:

package node

import "github.com/jonatan5524/own-kubernetes/pkg/agent/api"

const (
    NODE_NAME          = "node"
    NODE_IMAGE         = "own-kube-node"
    NODE_PORT          = api.PORT + "/tcp"
    NODE_PORT_HOST_IP  = "0.0.0.0"
    MEMORY_LIMIT       = 2.9e+9 // 2900MB
    CPU_LIMIT          = 2
    NODE_HOST_MIN_PORT = 10250
    NODE_HOST_MAX_PORT = 10300
)

All the constraints will be used in the following files and all the values are taken from the last article when we ran the docker container node.
In the pkg/node.go we define a Node struct:

type Node struct {
    Id   string
    Port string
}

Id - generated id for the node
Port - generate unused port for the node
In this file we also define function for creating a new node using the docker sdk:

func NewNode(cli *client.Client, ctx context.Context) (*Node, error) {
    exists, err := isNodeImageExists(cli, ctx)
    if err != nil {
        return nil, err
    } else if exists == false {
        return nil, fmt.Errorf("node image: %s not exists locally, need to build the image", NODE_IMAGE)
    }

    id := pkg.GenerateNewID(NODE_NAME)
    config := &container.Config{
        Image: NODE_IMAGE,
    }

    hostConfig := &container.HostConfig{
        PortBindings: nat.PortMap{
            nat.Port(fmt.Sprintf("%s/tcp", api.PORT)): []nat.PortBinding{
                {
                    HostIP:   NODE_PORT_HOST_IP,
                    HostPort: "0",
                },
            },
        },
        Resources: container.Resources{
            Memory:    MEMORY_LIMIT,
            CPUShares: CPU_LIMIT,
        },
        Privileged: true,
    }

    _, err = cli.ContainerCreate(ctx, config, hostConfig, nil, nil, id)
    if err != nil {
        return nil, err
    }

    return &Node{Id: id}, nil
}

We first check if the node image exists, then we generate an id, and create some config for the container:

node image
port forwarding from the port inside the container (the agent API) to outside of the container. The HostPort is 0 because Linux assigns a random open port when a service assigns a 0 port.
memory limit
CPU limit
privileged All the configurations are equivalent to the command we ran in the previous article:

sudo docker run -it --memory="2900MB" --cpus="2" -p 10250:10250 --privileged --name test --rm containerd_test

Then we create the container with the config and generated Id and return a new Node struct.
The isNodeImageExists function implementation:

func isNodeImageExists(cli *client.Client, ctx context.Context) (bool, error) {
    images, err := cli.ImageList(ctx, types.ImageListOptions{})
    if err != nil {
        return false, err
    }

    for _, image := range images {
        if strings.Contains(image.RepoTags[0], NODE_IMAGE) {
            return true, nil
        }
    }

    return false, nil
}

The function List the existing images, go through each one, and check if its RepoTag contains an own-kube-node

Let's move to pkg/node/service.go, the first function we will go through will be the initDockerConnection which initialized the docker client and context:

func initDockerConnection() (*client.Client, context.Context, error) {
    ctx := context.Background()
    cli, err := client.NewClientWithOpts(client.FromEnv, client.WithAPIVersionNegotiation())
    if err != nil {
        return nil, nil, err
    }

    return cli, ctx, nil
}

Next, is the NewNodeAndRun function which creates new node and starts the container:

func NewNodeAndRun() (*Node, error) {
    cli, ctx, err := initDockerConnection()
    if err != nil {
        return nil, err
    }
    defer cli.Close()

    node, err := NewNode(cli, ctx)
    if err != nil {
        return nil, err
    }

    if err := cli.ContainerStart(ctx, node.Id, types.ContainerStartOptions{}); err != nil {
        return nil, err
    }

    log.Printf("node created: %s\n", node.Id)
    log.Printf("starting node\n")

    container, err := cli.ContainerInspect(ctx, node.Id)
    if err != nil {
        return nil, err
    }

    // get linux generated port
    node.Port = container.NetworkSettings.Ports[nat.Port(fmt.Sprintf("%s/tcp", api.PORT))][0].HostPort

    log.Printf("node assign port: %s\n", node.Port)

    return node, nil
}

As you can see in this function, we are initializing the connection to the docker daemon, creating a new node and container, starting the container, and then because we assign port 0 we need to retrieve the assigned port by the system.

The next function is ListRunningNodes:

func ListRunningNodes() ([]string, error) {
    cli, ctx, err := initDockerConnection()
    if err != nil {
        return []string{}, err
    }
    defer cli.Close()

    runningNodes := []string{}

    filter := filters.NewArgs(filters.KeyValuePair{Key: "ancestor", Value: NODE_IMAGE})
    containers, err := cli.ContainerList(ctx, types.ContainerListOptions{Filters: filter})
    if err != nil {
        return runningNodes, err
    }

    for _, container := range containers {
        runningNodes = append(runningNodes, container.Names[0])
    }

    return runningNodes, nil
}

The function initializes the connection, calls the ContainerList method with the ancestor filter which means the image we want to filter, and then runs over each container to create a list of names.

The last function is KillNode function which simply stops the container and removes it:

func KillNode(name string) (string, error) {
    cli, ctx, err := initDockerConnection()
    if err != nil {
        return "", err
    }
    defer cli.Close()

    if err := cli.ContainerStop(ctx, name, nil); err != nil {
        return "", err
    }

    removeOptions := types.ContainerRemoveOptions{
        RemoveVolumes: true,
        Force:         true,
    }

    if err := cli.ContainerRemove(ctx, name, removeOptions); err != nil {
        return "", err
    }

    return name, nil
}

The last part we need to complete is cmd/node.go its structure is the same as cmd/pod.go, I'm not going to add this file here as this article is too long, you can see it in the source code.

we can see that everything works on the terminal:

❯ make
go build -o bin/agent pkg/agent/agent.go
sudo docker build -t own-kube-node .
...
❯ sudo ./bin/main node create
2022/10/12 19:18:26 node created: node-7daf80ec-0a46-4977-952c-66deb81e32f7
2022/10/12 19:18:26 starting node
2022/10/12 19:18:26 node assign port: 49167
❯ sudo docker ps
CONTAINER ID   IMAGE           COMMAND     CREATED         STATUS         PORTS                      NAMES
2066e8e0b933   own-kube-node   "./agent"   5 seconds ago   Up 4 seconds   0.0.0.0:49167->10250/tcp   node-7daf80ec-0a46-4977-952c-66deb81e32f
❯ curl -X POST localhost:49167/pods -H 'Content-Type: application/json' -d '{"name": "redis", "image registry": "docker.io/library/redis:alpine"}'
{"image registry":"docker.io/library/redis:alpine","name":"redis-a1cfce57-5db9-4894-a8cf-c277444e1b0c"}

And that's it, we have a node created by command using an API call that can also create for us a pod inside the node!

In the next article, we will cover some Linux network and get working communication with our pods and nodes.

As always, the source code can be found here, the changes were in pkg/node, Dockerfile, Makefile, cmd/node.

Build own Kubernetes - Node setup

Jonatan Ezron — Tue, 11 Oct 2022 15:20:19 +0000

An updated article on the above can be found in Medium: https://medium.com/@jonatan5524/a-journey-of-writing-my-own-kubernetes-ef45839a769d

In the last posts we handled pods, Now in this article, we are going to focus on Node.
But first, what is a node?
Node can be a physical or virtual machine. There is a master node which contains the control plane and etcd and worker nodes which contains the running pods, kubelet, and k-proxy. In this article, we will focus on the worker node creation.

Before this article I had some refactor the code, The creating and running methods of the pods, output set to log file and not STDIO, added a new method to get the pod logs, and the create pod commands now calls a NewPodAndRun function which creates and runs the code.

I was struggling to find the right platform for the node os, at first I looked for an appropriate docker container image that will run the agent (kubelet) and containerd service in the background but with no luck. So we need to build some base image on our own, this node VM can also be created with KVM, I decided docker for the easier approach.
We first need to build a Dockerfile with Golang, we will use the base image ubuntu (I have tried alpine but had some troubles building the go source code 😔 ), install go and containerd, to check if everything works I have taken a simple HTTP server:

package main

import (
    "fmt"
    "net/http"
    "log"
    "os"
    "os/exec"
)

func hello(w http.ResponseWriter, req *http.Request) {

    fmt.Fprintf(w, "hello\n")
}

func headers(w http.ResponseWriter, req *http.Request) {

    for name, headers := range req.Header {
        for _, h := range headers {
            fmt.Fprintf(w, "%v: %v\n", name, h)
        }
    }
}

func startContainerd() {
    cmd := exec.Command("containerd")
    cmd.Stdout = os.Stdout
    err := cmd.Start()
    if err != nil {
        log.Fatal(err)
    }
    log.Printf("just ran subprocess %d", cmd.Process.Pid)
}

func main() {
    startContainerd()

    http.HandleFunc("/hello", hello)
    http.HandleFunc("/headers", headers)

    http.ListenAndServe(":8090", nil)
}

The server first starts a containerd service process and then starts the server.
For the Dockerfile we installed Go, containerd, and copy and build or main.go:

FROM ubuntu

WORKDIR /agent

RUN apt-get update \
    && apt-get install -y wget git gcc \
    && wget -P /tmp https://go.dev/dl/go1.19.2.linux-amd64.tar.gz \
    && tar -C /usr/local -xzf "/tmp/go1.19.2.linux-amd64.tar.gz"

ENV GOPATH /go
ENV PATH $GOPATH/bin:/usr/local/go/bin:$PATH
RUN mkdir -p "$GOPATH/src" "$GOPATH/bin" && chmod -R 777 "$GOPATH"

COPY main.go .

RUN go build -o main main.go

RUN apt-get install -y containerd

EXPOSE 8090

ENTRYPOINT [ "./main" ]

We build and run with enlarged CPU and memory limit (the sizes for the CPU and memory I have got from minikube implementation) so we have enough for the pods we will create inside:

❯ sudo docker build . -t containerd_test
❯ sudo docker run -it --memory="2900MB" --cpus="2" -p 8090:8090 containerd_test
❯ sudo docker ps
CONTAINER ID   IMAGE             COMMAND    CREATED          STATUS          PORTS                                       NAMES
e26fd47de621   containerd_test   "./main"   25 seconds ago   Up 24 seconds   0.0.0.0:8090->8090/tcp, :::8090->8090/tcp   reverent_solomon
❯ sudo docker exec e26 ctr c ls
CONTAINER    IMAGE    RUNTIME

As you can see the HTTP server is running and there is a containerd process running as well.

Now we need to build our agent, the base functionality of creating pods is already set, we just need some API endpoints.
We will use the echo framework for the REST API, to implement the different API endpoints: POST for creating pods, DELETE for deleting, GET for getting the pods, and for getting pod logs.
Let's start with the main function of the agent binary, it will be in pkg/agent/agent.go that will start the containerd process and config the echo REST API:

package main

import (
    "fmt"
    "log"
    "os"
    "os/exec"

    "github.com/jonatan5524/own-kubernetes/pkg/agent/api"
    "github.com/labstack/echo/v4"
    "github.com/labstack/echo/v4/middleware"
)

func initRoutes(e *echo.Echo) {
    e.POST("/pod", api.CreatePod)
    e.GET("/pod/:id/log", api.LogPod)
    e.GET("/pod", api.GetAllPods)
    e.DELETE("/pod/:id", api.DeletePod)
}

func initMiddlewares(e *echo.Echo) {
    e.Use(middleware.LoggerWithConfig(middleware.LoggerConfig{
        Format: "method=${method}, uri=${uri}, status=${status}\n",
    }))

    e.HTTPErrorHandler = func(err error, c echo.Context) {
        c.Logger().Error(err)

        e.DefaultHTTPErrorHandler(err, c)
    }
}

func startContainerd() {
    cmd := exec.Command("containerd")
    cmd.Stdout = os.Stdout
    err := cmd.Start()
    if err != nil {
        log.Fatal(err)
    }
    log.Printf("containerd run on %d", cmd.Process.Pid)
}

func main() {
    startContainerd()

    e := echo.New()

    initMiddlewares(e)
    initRoutes(e)

    e.Logger.Fatal(e.Start(fmt.Sprintf(":%s", api.PORT)))
}

We used port 10250 for the agent API because this is the same port used in the Kubernetes agent.
Next is the pkg/agent/api/pod.go that will contains the different functions that will handle the pod endpoint:

package api

import (
    "net/http"

    "github.com/jonatan5524/own-kubernetes/pkg/pod"
    "github.com/labstack/echo/v4"
)

type podDTO struct {
    ImageRegistry string `json:"image registry"`
    Name          string `json:"name"`
}

func CreatePod(c echo.Context) error {
    podDto := new(podDTO)
    if err := c.Bind(podDto); err != nil {
        return err
    }

    id, err := pod.NewPodAndRun(podDto.ImageRegistry, podDto.Name)
    if err != nil {
        return err
    }

    return c.JSON(http.StatusCreated, podDTO{
        ImageRegistry: podDto.ImageRegistry,
        Name:          id,
    })
}

func LogPod(c echo.Context) error {
    logs, err := pod.LogPod(c.Param("id"))
    if err != nil {
        return err
    }

    return c.String(http.StatusOK, logs)
}

func GetAllPods(c echo.Context) error {
    pods, err := pod.ListRunningPods()
    if err != nil {
        return err
    }

    return c.JSON(http.StatusCreated, pods)
}

func DeletePod(c echo.Context) error {
    if _, err := pod.KillPod(c.Param("id")); err != nil {
        return err
    }

    return c.NoContent(http.StatusNoContent)

}

And as mentioned above, we are creating a node image of the node container, the Dockerfile will be located in the project root so we can copy all the project folders:

# Dockerfile for node image
FROM ubuntu

WORKDIR /agent

RUN apt-get update \
    && apt-get install -y wget git gcc \
    && wget -P /tmp https://go.dev/dl/go1.19.2.linux-amd64.tar.gz \
    && tar -C /usr/local -xzf "/tmp/go1.19.2.linux-amd64.tar.gz"

ENV GOPATH /go
ENV PATH $GOPATH/bin:/usr/local/go/bin:$PATH
RUN mkdir -p "$GOPATH/src" "$GOPATH/bin" && chmod -R 777 "$GOPATH"

RUN apt-get install -y containerd

COPY go.mod .
COPY go.sum .
RUN go mod download  

COPY . .

RUN go build -o main pkg/agent/agent.go

EXPOSE 8090

ENTRYPOINT [ "./main" ]

Let's build and test our work!
We build and run our container node with the previous requirements::

❯ sudo docker build -t containerd_test .
❯ sudo docker run -it --memory="2900MB" --cpus="2" -p 10250:10250 --privileged --name test --rm containerd_test

And now if we send a request to create a Redis pod:

❯ curl -X POST localhost:10250/pod -H 'Content-Type: application/json' -d '{"name": "redis", "image registry": "docker.io/library/redis:alpine"}'
{"image registry":"docker.io/library/redis:alpine","name":"redis-c5ad79db-0e8e-4526-a9d1-366cefd0e9d5"}

The pod is created!
In the agent logs:

2022/10/10 15:36:07 pod created: redis-c5ad79db-0e8e-4526-a9d1-366cefd0e9d5
2022/10/10 15:36:07 starting po

We can test and see that also all the other endpoints works:

❯ curl localhost:10250/pod
["redis-c5ad79db-0e8e-4526-a9d1-366cefd0e9d5"]

❯ curl localhost:10250/pod/redis-c5ad79db-0e8e-4526-a9d1-366cefd0e9d5/log
1:C 10 Oct 2022 15:36:07.757 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 10 Oct 2022 15:36:07.757 # Redis version=7.0.5, bits=64, commit=00000000, modified=0, pid=1, just started
...
❯ curl -X DELETE  localhost:10250/pod/redis-c5ad79db-0e8e-4526-a9d1-366cefd0e9d5

Everything works!

On the next article we will focus on automating the node creation and other methods on node with commands like the pods.

The full source code can be found here, the changes were in pkg/agent and Dockerfile.

Build own Kubernetes - Pods listing and deletion

Jonatan Ezron — Fri, 07 Oct 2022 14:05:43 +0000

An updated article on the above can be found in Medium: https://medium.com/@jonatan5524/a-journey-of-writing-my-own-kubernetes-ef45839a769d

In the previous article, we created and run the pods, in this article we will list and delete pods on command.

Before this article I had some refactor the code, The methods related to Pod and RunningPod structs are in pkg/pod/pod.go and access functions to run operations related to pods like creating a new pod is in pkg/pod/service.go

We want to List all the running pods, we implement a simple function in pkg/pod/service.go to list all the existing tasks:

func initContainerdConnection() (*containerd.Client, context.Context, error) {
    client, err := containerd.New(SOCKET_PATH)
    if err != nil {
        return nil, nil, err
    }

    ctx := namespaces.WithNamespace(context.Background(), NAMESPACE)

    return client, ctx, nil
}

func ListRunningPods() ([]string, error) {
    client, ctx, err := initContainerdConnection()
    if err != nil {
        return nil, err
    }

    runningPods := []string{}

    containers, err := client.Containers(ctx)
    if err != nil {
        return runningPods, err
    }

    for _, container := range containers {
        _, err = container.Task(ctx, cio.Load)

        if err == nil {
            runningPods = append(runningPods, container.ID())
        }
    }

    return runningPods, nil
}

The initContainerdConnection function was created on the refactor phase, this function is called to create a client and ctx to make operations on containerd.
The function goes through every existing container and tries to Load to it, if no error returns this means there is a running task on the container, meaning the pod is running.

In the cmd/pod.go

var listCmd = &cobra.Command{
    Use:   "list",
    Short: "lists existing pods",
    RunE: func(cmd *cobra.Command, args []string) error {
        runningPods, err := pod.ListRunningPods()
        if err != nil {
            return err
        }

        for _, pod := range runningPods {
            fmt.Println(pod)
        }

        return nil
    },
}

Let's check (for this I commented the kill and deletion of the pod creation):

❯ sudo ./main pod create --registry docker.io/library/redis:alpine --name redis
pod created: redis-9079a2e3-87a0-4a08-8b96-14926b6edb45
starting pod
pod started: redis-9079a2e3-87a0-4a08-8b96-14926b6edb45
❯ sudo ./main pod list
redis-9079a2e3-87a0-4a08-8b96-14926b6edb45

great, It works!

Now let's focus on the delete command, we want the delete to accept a pod id and delete it, it prints the deleted pod id.
So in cmd/pkd/pod.go:

var killCmd = &cobra.Command{
    Use:   "kill",
    Short: "kill existing pod",
    RunE: func(cmd *cobra.Command, args []string) error {
        id, err := pod.KillPod(name)
        if err != nil {
            return err
        }

        fmt.Println(id)

        return nil
    },
}

func init() {
.
.
.

    podCmd.AddCommand(killCmd)
    killCmd.Flags().StringVar(&name, "id", "", "the pod id (required)")
    killCmd.MarkFlagRequired("id")
}

Now in the pkg/pod/service.go, we add a KillPod function, which searches for a container and container's task, creates from it a Pod and RunningPod struct, and executes the Kill and Delete methods. The creation of the structs is for the kill and delete to be handled properly.

func KillPod(name string) (string, error) {
    client, ctx, err := initContainerdConnection()
    if err != nil {
        return "", err
    }

    container, err := client.LoadContainer(ctx, name)
    if err != nil {
        return "", err
    }

    task, err := container.Task(ctx, cio.Load)
    if err != nil {
        return "", err
    }

    exitStatusC, err := task.Wait(ctx)
    if err != nil {
        return "", err
    }

    runningPod := RunningPod{
        task: &task,
        Pod: &Pod{
            client:    client,
            ctx:       &ctx,
            container: &container,
            Id:        name,
        },
        exitStatusC: exitStatusC,
    }

    _, err = runningPod.Kill()
    if err != nil {
        return "", err
    }

    runningPod.Pod.Delete()

    return name, nil
}

Let's test our changes:

❯ sudo ./main pod create --registry docker.io/library/redis:alpine --name redis
pod created: redis-3f6f15fb-ff52-4d61-937f-db01d02f1b61
starting pod
❯ sudo ./main pod list
redis-3f6f15fb-ff52-4d61-937f-db01d02f1b61
❯ sudo ./main pod kill --id redis-3f6f15fb-ff52-4d61-937f-db01d02f1b61
redis-3f6f15fb-ff52-4d61-937f-db01d02f1b61
❯ sudo ./main pod list

Great, it works!

On the next article we will focus on Node setup and agent.

The full source code can be found here, the changes were in pkg/pod/service.go and cmd/pod.go.

Build own Kubernetes - Pods creation

Jonatan Ezron — Fri, 07 Oct 2022 14:02:55 +0000

An updated article on the above can be found in Medium: https://medium.com/@jonatan5524/a-journey-of-writing-my-own-kubernetes-ef45839a769d

At first, we will focus on the smallest unit of Kubernetes - Pod, this chapter will focus on creating and running a pod. Each container will run in containerd (you need to install containerd in your environment) and will be managed through it.

At the start of this project, we will implement basic tasks with cmd commands using Cobra, and later we will proceed to more advanced stuff.

We initialize the project using cobra-cli and add new commands pod for pods command, list for listing existing pods, create for creating and running new pods (for now).

cobra-cli init
cobra-cli add pod
cobra-cli add list
cobra-cli add create

We will move the commands to a single pod file command inside the cmd file so that we can execute pod list or pod create:

// cmd/pod.go
package cmd

import (
    "github.com/spf13/cobra"
)

var podCmd = &cobra.Command{
    Use:   "pod",
    Short: "The command line tool to run commands on pods",
}

var createCmd = &cobra.Command{
    Use:   "create",
    Short: "Create new pod",
    // Run: ,
}

var listCmd = &cobra.Command{
    Use:   "list",
    Short: "lists existing pods",
    // Run: ,
}

func init() {
    rootCmd.AddCommand(podCmd)
    podCmd.AddCommand(listCmd)
    podCmd.AddCommand(createCmd)
}

Let's implement a pod creation in pkg/pod/pod.go for now we only support 1 container on 1 pod.
We are using containerd, so make sure it is running on your system.
Source on how to use the containerd package.
We will define a few structs. the first two would be: a Pod struct which will represent an existing Pod instance in containerd, and a RunningPod struct which will represent a pod instance that currently running.

type Pod struct {
    Id        string
    client    *containerd.Client
    ctx       *context.Context
    container *containerd.Container
}

Id - generated id for the pod
client - the containerd client to communicate with
ctx - context to use, will calls to client methods
container - the pod's container instance

type RunningPod struct {
    Pod         *Pod
    task        *containerd.Task
    exitStatusC <-chan containerd.ExitStatus
}

Pod - the configured Pod instance which was created from
task - the current process task that is running
exitStatusC - channel to get the exit status from

Now after we defined the structs let's create a NewPod method for Pod creation, in the following function we create a new containerd client, a new context, pull the image, generate a new id, and create a new container:

func NewPod(registryImage string, name string) (*Pod, error) {
    client, err := containerd.New("/run/containerd/containerd.sock")
    if err != nil {
        return nil, err
    }

    ctx := namespaces.WithNamespace(context.Background(), "own-kubernetes")

    image, err := client.Pull(ctx, registryImage, containerd.WithPullUnpack)
    if err != nil {
        return nil, err
    }

    id := generateNewID(name)

    container, err := client.NewContainer(
        ctx,
        id,
        containerd.WithImage(image),
        containerd.WithNewSnapshot(id+"-snapshot", image),
        containerd.WithNewSpec(oci.WithImageConfig(image)),
    )
    if err != nil {
        return nil, err
    }

    return &Pod{
        Id:        id,
        container: &container,
        ctx:       &ctx,
        client:    client,
    }, nil
}

func generateNewID(name string) string {
    id := uuid.New()

    return fmt.Sprintf("%s-%s", name, id)
}

The id generation is generated with the google uuid package and with the name given as seen above.

Next, we implement the Run pod method, at first we create a new task which will be a process of a running pod, wait for the creation and start the task:

func (pod *Pod) Run() (*RunningPod, error) {
    task, err := (*pod.container).NewTask(*pod.ctx, cio.NewCreator(cio.WithStdio))
    if err != nil {
        return nil, err
    }

    exitStatusC, err := task.Wait(*pod.ctx)
    if err != nil {
        fmt.Println(err)
    }

    if err := task.Start(*pod.ctx); err != nil {
        return nil, err
    }

    return &RunningPod{
        Pod:         pod,
        task:        &task,
        exitStatusC: exitStatusC,
    }, nil
}

Next, we implement the Kill method for the running pod, it will kill the existing process, delete the existing task, and returns the status code. And implement the Delete method for an existing pod which will delete the existing container and close the client connection

func (pod *RunningPod) Kill() (uint32, error) {
    // kill the process and get the exit status
    if err := (*pod.task).Kill(*pod.Pod.ctx, syscall.SIGTERM); err != nil {
        return 0, err
    }

    // wait for the process to fully exit and print out the exit status
    status := <-pod.exitStatusC
    code, _, err := status.Result()
    if err != nil {
        return 0, err
    }

    (*pod.task).Delete(*pod.Pod.ctx)

    return code, nil
}

func (pod *Pod) Delete() {
    (*pod.container).Delete(*pod.ctx, containerd.WithSnapshotCleanup)
    pod.client.Close()
}

On cmd/pod.go we implement a simple command for creating and running a pod, we define the flags --registry and --name:

func init() {
    rootCmd.AddCommand(podCmd)
    podCmd.AddCommand(listCmd)

    podCmd.AddCommand(createCmd)
    createCmd.Flags().StringVar(&imageRegistry, "registry", "", "image registry to pull (required)")
    createCmd.MarkFlagRequired("registry")
    createCmd.Flags().StringVar(&name, "name", "nameless", "the pod name")
}

and created a simple implementation for creating and running the pod and after 3 seconds kill it and delete:

var (
    imageRegistry string
    name          string
)
var createCmd = &cobra.Command{
    Use:   "create",
    Short: "Create new pod",
    RunE: func(cmd *cobra.Command, args []string) error {
        pod, err := pod.NewPod(imageRegistry, name)
        if err != nil {
            return err
        }

        fmt.Printf("pod created: %s\n", pod.Id)
        fmt.Printf("starting pod\n")

        runningPod, err := pod.Run()
        if err != nil {
            return err
        }

        fmt.Printf("pod started: %s\n", pod.Id)

        time.Sleep(3 * time.Second)

        fmt.Printf("killing pod\n")

        code, err := runningPod.Kill()
        if err != nil {
            return err
        }
        fmt.Printf("pod killed: %s\n", pod.Id)

        fmt.Printf("%s exited with status: %d\n", runningPod.Pod.Id, code)

        pod.Delete()

        fmt.Printf("container deleted: %s\n", pod.Id)

        return nil
    },
}

So we implemented all the necessary things to create and run a pod, let's see it running! (make sure the containerd is running)
In the terminal build the go project:
go build main.go
and let's create a new Redis pod:

❯ sudo ./main pod create --registry docker.io/library/redis:alpine --name redis
pod created: redis-73c4d234-2fe1-4b8f-bfe4-aa9044dc064a
starting pod
pod started: redis-73c4d234-2fe1-4b8f-bfe4-aa9044dc064a
1:C 07 Oct 2022 10:53:31.804 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 07 Oct 2022 10:53:31.804 # Redis version=7.0.5, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 07 Oct 2022 10:53:31.804 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
1:M 07 Oct 2022 10:53:31.805 # You requested maxclients of 10000 requiring at least 10032 max file descriptors.
1:M 07 Oct 2022 10:53:31.805 # Server can't set maximum open files to 10032 because of OS error: Operation not permitted.
1:M 07 Oct 2022 10:53:31.805 # Current maximum open files is 1024. maxclients has been reduced to 992 to compensate for low ulimit. If you need higher maxclients increase 'ulimit -n'.
1:M 07 Oct 2022 10:53:31.805 * monotonic clock: POSIX clock_gettime
1:M 07 Oct 2022 10:53:31.805 * Running mode=standalone, port=6379.
1:M 07 Oct 2022 10:53:31.805 # Server initialized
1:M 07 Oct 2022 10:53:31.805 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
1:M 07 Oct 2022 10:53:31.806 * Ready to accept connections
killing pod
1:signal-handler (1665140014) Received SIGTERM scheduling shutdown...
1:M 07 Oct 2022 10:53:34.816 # User requested shutdown...
1:M 07 Oct 2022 10:53:34.816 * Saving the final RDB snapshot before exiting.
1:M 07 Oct 2022 10:53:34.895 * DB saved on disk
1:M 07 Oct 2022 10:53:34.895 # Redis is now ready to exit, bye bye...
pod killed: redis-73c4d234-2fe1-4b8f-bfe4-aa9044dc064a
redis-73c4d234-2fe1-4b8f-bfe4-aa9044dc064a exited with status: 0
container deleted: redis-73c4d234-2fe1-4b8f-bfe4-aa9044dc064a

It works!

On the next article we will list and delete pods on command.

The full source code can be found here, the changes were in pkg/pod/service.go and cmd/pod.go.

Build my own Kubernetes journey

Jonatan Ezron — Fri, 07 Oct 2022 13:19:03 +0000

An updated article on the above can be found in Medium: https://medium.com/@jonatan5524/a-journey-of-writing-my-own-kubernetes-ef45839a769d

I was fond of container orchestration, and in particular of course Kubernetes. I wanted to go into the Kubernetes open source, When I saw the code base I was initially overwhelmed.

I decided with my love for Golang I want to build it on my own to understand how It works better and maybe get into its code base more easily.
These multiple articles will go through my journey on how to build Kubernetes on my own with Golang.

Disclaimer: This project will be a minimal but working implementation, not close to the official Kubernetes implementation and maybe a lot different, but trying to follow its requirements of container orchestration, and all of it will be locally on VMs and not distributed over multiple servers.

Each article will cover some topics and source code on the implementation.
Before we begin I welcome of course throughout my articles to suggest fixes in the comments (including typos because English isn't my native language 😉).

What do you need to know before starting this series?

basic Go syntax
basic docker concepts
basic understanding in Linux

How Kubernetes works

This article will focus on how Kubernetes works on the inside, if you are already familiar with the inside of Kubernetes you can go to the next article.
source for this section

First, let's define a container orchestration system - a way to manage the lifecycle of containerized applications, like the ability to automate deployment and scale easily. These containers act as replicas and serve to load balance incoming requests. A container orchestrator supervises these groups of containers and ensures that everything works correctly.

We will break down each Kubernetes concept we should know

Pod

The smallest unit that Kubernetes administers, can contain one or more containers. Pods have a single IP address that is assigned to every container within the pod.

Deployment

Define the way at which you want to run your application by letting you set the details of your pods, like replicas, update strategy, etc. It will describe to Kubernetes your application's desired state.

Service

Abstraction over the pods, the interface the various application consumers interact with because pod states are always changing and cant be relied on.

Node

Manages and runs pods, it's a machine (virtual or physical) that performs the given work.

Control Plane

The main entry point of the administrators and users to manage the various nodes.

Cluster

All the above components put together as a single unit.

API Server

Exposes a REST interface to the Kubernetes cluster. All operations on pods, services, etc. are executed with the API endpoints provided by it.

Scheduler

Responsible for assigning work to various nodes, keeping watch over the resource capacity, and ensuring that a worker node's performance is within an appropriate threshold.

Controller Manager

Making sure that the state of the cluster components works as expected, oversees various controllers which respond to events, like when a node goes down.

Kubelet

Tracks the state of a pod to ensure that all the containers are running. It provides a heartbeat message every few seconds to the control plane.

Kube proxy

Routes the traffic coming into a node from the service into the correct containers.

etcd

Distributed key-value store that Kubernetes uses to share information about the overall state of a cluster.

Complete architecture of kubernetes

In the next chapter we will get into the code, will cover pod creation and running with containerd.