Forem: Muaaz Saleem

3 Principles I wanna apply at work in 2021

Muaaz Saleem — Sun, 17 Jan 2021 17:40:17 +0000

I'm a big fan of goals. They can be really motivating, especially when I have a lot of control over how I go about achieving them. I've been struggling with having productive S.M.A.R.T goals at work though.

So this year I'm trying out something new. Instead of any specific goals, I hope to apply these three principles to the challenges that I come upon professionally.

Accept the Chaos

A job as a Site Reliability Engineer is an adventure at best and total mayhem on the, not so best, days. There are of course, ideals of minimizing interruptions and reliable planning that we embrace but I think it's healthy to accept that at least some chaos is just part of the job.

Embrace Spontaneity

Sometimes I hear about a project or an idea and I just completely fall in love with it. I just can't shake its sheer possibility. This year, I wanna let myself do that more often! If some chaos comes with the gig then making the best of an exciting opportunity seems like the best way to grow.

Reduce Work In Progress

After we begin a project and before we ship it to our users, there's work in progress. Every project is a work in progress for a period, some for longer than others.

Some Work In Progress is good for me! It means we're working on non-trivial stuff. It also helps me to have a few things in progress, so when I'm stuck on or bored/tired of a project for the day, I can switch to another fairly productive thing.

Too much work in progress is bad for me! I have to switch between projects too often and I usually don't make enough progress on any single one.

Excited to look back

I'm excited to look back at this commitment and see the effect that it would've had on my professional life!

Pros and Cons of Using client-go in Kubernetes controllers

Muaaz Saleem — Sun, 17 Jan 2021 17:38:21 +0000

When writing a kubernetes controller I often use client-go to talk to the Kubernetes API. I'm beginning to realize though, that client-go isn't always the best approach, sometimes it can be simpler to use a hand rolled HTTP client, based on go's net/http package.

Today, I want to share some of the Pros & Cons of using client-go, as I understand them, in Kubernetes controllers or other go applications.

Pros of using client-go

Helpful Utilities

client-go comes with helpful utilities e.g to create in-cluster or out-of-cluster kubeconfig and authenticate against the API Server i.e kubernetes.NewForConfig and clientcmd.BuildConfigFromFlags respectively.

I end up using these in every Kubernetes app that I write.

Implementations for all kubernetes API Objects

I find unmarshalling kubernetes resources from JSON to go, pretty cumbersome, particularly if I need to access a deeply nested field e.g reading the replicas from the status of a Deployment object.

client-go handles marhsalling/unmarshalling of all compatible Kubernetes resources out of the box. So that's a big help.

Powerful abstractions for common use casses

Common use cases such as watching resources or testing API calls are well supported with abstractions like Shared Informers and fake-client.

Cons of using client-go

Cognitive Overload

This by far the biggest frustration I have when using client-go. Just figuring out the right k8s.io/api & k8s.io/api-machinery dependencies to match my k8s.io/client-go version used to be a struggle. Thankfully with go modules and the new versioning scheme that's really improved.

I still get lost in all the imports regularly, with the sheer no. of structs and interfaces, it's not always clear if I actually need everything I'm importing. Integration tests written with the fake client can also be daunting for a new person to understand.

Sometimes this is just symptom of the scarce docs availble!

Scarce Docs

Even though there are now actually some hello-world examples in the client-go repo, docs for the go client are still pretty scarce. The first time I had to learn what Informers were, my choices? A random blog post or the actual code.

This means that my usage of the library is only as good as the latest blogpost/book/code I've read. I can understand feeling that way about a more complicated tool, like a Database, but should I have to feel this way about an HTTP client?

Not the most idiomatic go-code

Using client-go sometimes requires comitting generated code or dealing with *string types. The library isn't always an example of the most idomatic go code.

This might be totally OK for experience go devs, but as someone still learning how to write good go code. I find this confusing.

Further Resources

I still think client-go is worth using for full fledge controllers, especially when dealing with multiple Kubernetes objects. For simple scripts, I'd like to invest more in just using plain go.

If you'd like to learn more about client-go, I can't recommend Programming Kubernetes by Michael Hausenblas and Stefan Schimanski enough!

Some other places:

Creating a Go Kubernetes client with client-go

Muaaz Saleem — Sun, 17 Jan 2021 17:35:50 +0000

I mostly use client-go to talk to the Kubernetes API in my go applications.

Whenever I'm starting a new project I inevitably have to figure out the setup steps. I thought I'll just document these for myself here.

A `client-go` "Hello, World!"

Creating a client that implements kubernetes.Interface is just kubernetes.NewForConfig()

import "k8s.io/client-go/kubernetes"

client, err := kubernetes.NewForConfig(kubeconfig)
if err != nil {
    return nil, fmt.Errorf("unable to create a client: %v", err)
}

Creating Client Config

Creating the config defers for go clients within and outside of the clients. I think this is mostly because access tokens need to be read to from a mounted file rather than the kubeconfig file.

Creating out of cluster config

import "k8s.io/client-go/tools/clientcmd"

config, err := clientcmd.BuildConfigFromFlags(
    "", kubeconfigPath,
)
if err != nil {
    return nil, fmt.Errorf(
        "unable to load kubeconfig from %s: %v", 
        kubeconfigPath, err,
        )
}

Creating in cluster config

import "k8s.io/client-go/rest"

config, err := rest.InClusterConfig()
if err != nil {
    return nil, fmt.Errorf("unable to load in-cluster config: %v", err)
}

Putting it all together

import (
    "fmt"
    "k8s.io/client-go/kubernetes"
    "k8s.io/client-go/rest"
    "k8s.io/client-go/tools/clientcmd"
)

func createClient(kubeconfigPath string) (kubernetes.Interface, error) {
    var kubeconfig *rest.Config

    if kubeconfigPath != "" {
        config, err := clientcmd.BuildConfigFromFlags("", kubeconfigPath)
        if err != nil {
            return nil, fmt.Errorf("unable to load kubeconfig from %s: %v", kubeconfigPath, err)
        }
        kubeconfig = config
    } else {
        config, err := rest.InClusterConfig()
        if err != nil {
            return nil, fmt.Errorf("unable to load in-cluster config: %v", err)
        }
        kubeconfig = config
    }

    client, err := kubernetes.NewForConfig(kubeconfig)
    if err != nil {
        return nil, fmt.Errorf("unable to create a client: %v", err)
    }

    return client, nil
}

Testing with Fake Client

When testing client-go code it's useful to use the built-in fake client. Here's a short example:

import (
    "context"
    "testing"

    "k8s.io/client-go/kubernetes/fake"
    appsv1 "k8s.io/api/apps/v1"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "github.com/stretchr/testify/require"
)

func TestKubeClient(t *testing.T) {
    fc := fake.NewSimpleClientset(&exampleDeploy)

    deploy, err := fc.AppsV1().Deployments("ns").Get(
        context.Background(),
        "deployment",
        metav1.GetOptions{},
    )

    require.NoError(t, err)

Managing package dependencies

client-go version needs to match that of the cluster we're using. I have sometimes had trouble with importing the right k8s.io/api and k8s.io/apimachinery dependencies so here's a example for that as well.

go.mod


module github.com/muaazsaleem/example

go 1.13

require (
// we only really need client-go here the right `api` and 
// `apimachinery` versions "should" get picked up
    k8s.io/client-go v0.19.0
    k8s.io/api v0.19.0
    k8s.io/apimachinery v0.19.0
// stuff I use in pretty much all my projects
    github.com/sirupsen/logrus v1.7.0
    github.com/stretchr/testify v1.6.1
)

Further Resources

If you'd like to learn more about building things with the Kubernetes API, Vladimir Vivien has a whole series on it.

The client-go docs have also improved. There are examples now!

5 Time saving things about client-go, I didn't know!

Muaaz Saleem — Sun, 17 Jan 2021 17:34:06 +0000

client-go is the official go client for talking to Kubernetes (k8s) APIs.

A while back, I read parts of Programming Kubernetes by Michael Hausenblas & Stefan Schimanski. I learned a whole lot about client-go that still helps me.

k8s API is divided into apigroups! Core objects don’t (like pods), everything else is pretty much under an API Group
k8s.io/client-go isn’t enough to talk to kubernetes API, you need k8s.io/api and k8s.io/apimachinery too
- You have to match their versions for it to all work!
- See the client-go versioning for simple instructions!
Kinds map to resources, resources map to HTTP Paths through RESTMappers, Scheme connects a kind and a Resource
Almost always use informers instead of watch to watch events, the exception is just watching events to log them, then maybe it doesn’t matter.
- See the package docs
Last but not the least, there’s a fake client! You can use it to test your code!
- See example!

There are many working examples in the upsteam repo now! Check them out to learn more.

Creating a Test HTTP Server in Go

Muaaz Saleem — Sun, 17 Jan 2021 17:32:32 +0000

httptest.NewServer gives us the ability to spin up a local HTTP Server. This is very useful say, when testing an HTTP client.

Usage

Here's a very basic usage of httptest.NewServer:

server := httptest.NewServer(
        http.HandlerFunc(
            func(w http.ResponseWriter, req *http.Request) {
            _, _ = fmt.Fprint(w, "OK")
        }))
    defer server.Close()

In the code above, server.URL can be used to connect to the server. For example, we could pass it to the following pingService func to test it.

func pingService(url string) ServiceStatus {
    // TODO: make sure url is actually a valid URL
    res, err := http.Get(url)
    // expecting the HTTP client to err out if the service is down
    if err != nil {
        // TODO: add TestURL to the log
        log.Errorf("[%s], is DOWN", url)
        return ServiceDown
    }
}

A more complete example

Following is a more complete example of pingService and its tests:

`pingService()`

// ServiceStatus indicates the status of a service i.e "UP", "FAILING", "DOWN"
type ServiceStatus string

const (
    ServiceUP      = ServiceStatus("UP")
    ServiceDown    = ServiceStatus("DOWN")
    ServiceFailing = ServiceStatus("FAILING")
)

func pingService(server *http.Server) ServiceStatus {
    // TODO: make sure server.URL is actually a valid URL
    res, err := http.Get(server.URL)
    // expecting the HTTP client to err out if the service is down
    if err != nil {
        // TODO: add TestURL to the log
        log.Errorf("[%s], is DOWN", server.URL)
        return ServiceDown
    }

    if res.StatusCode >= 200 && res.StatusCode <= 500 {
        log.Infof("[%s], is UP", server.URL)
        return ServiceUP
    } else {
        log.Errorf("[%s], is FAILING", server.URL)
        return ServiceFailing
    }
}

Testing `pingService()`

// TestStatusDown tests pinging a non running HTTP service
func TestStatusDown(t *testing.T) {
    ss := pingService(exampleTestService)
    require.Equal(t, ServiceDown, ss)
}

// TestStatusUP tests a running HTTP service
func TestStatusUP(t *testing.T) {
    sv := httptest.NewServer(
        http.HandlerFunc(
            func(w http.ResponseWriter, req *http.Request) {
            _, _ = fmt.Fprint(w, "OK")
        }))
    defer sv.Close()

    ss := pingService(sv)
    require.Equal(t, ServiceUP, ss)
}

// TestStatusFailing tests a failing HTTP service
func TestStatusFailing(t *testing.T) {
    sv := httptest.NewServer(
        http.HandlerFunc(
            func(w http.ResponseWriter, req *http.Request) {
            w.WriteHeader(http.StatusInternalServerError)
        }))
    defer sv.Close()

    ss := pingService(sv)
    require.Equal(t, ServiceFailing, ss)
}

Further Resources:

The Three Main Flavors of Kubernetes related Engineering Roles

Muaaz Saleem — Sun, 17 Jan 2021 17:29:46 +0000

As Kubernetes continues to gain traction, Kubernetes related oppurtunities continue to grow as well. That's great! But while most Kubernetes related engineering roles are pretty similar to each other, in my observation, there seem to be important differences between them as well.

Here, I just want to capture the three main flavours Kubernetes related Engineering roles I've noticed and their pros and cons.

Site Reliability Engineer

Also known as DeveOps Engineer or Cloud Engineer, the Site Reliability Engineer role is the most common of the three we'll talk about today. The distinguishing sign of the role is that you're responsible for (a) production Kubernetes cluster(s) and your team is primarily measured by it's reliabilty, cost effectiveness and utility.

Google has written a lot about the role and different companies abide by that description to various degrees. This is also the only role of the three, that I have personal experience with.

Pros

Hands-on Experience operating (a) Kubernetes cluster(s) i.e
- Lots of Production Insights to share with the rest of the community
Incentive to influence the Cloud Native Community to solve important problems together
Lots of fun for folks who enjoy learning new things all the time
Lots of opportunities with the growing no. of businesses adopting Kubernetes

Cons

Time & Focus are divided between programming, operations & communication. Ergo:
- Limited chances to role the dice on long term software projects
Wide scope of respobsibility i.e:
- The whole cluster rather than just Ingress, although possible in a big enough team
Shifting respobsibilities as Cloud vendor capabilities grow/change
Employer may not have the bandwidth/budget for significant community participation

Software Engineer ( Kubernetes )

If you've ever wondered who maintains, the Kubernetes Core or important plugins like CoreDNS, it's Kubernetes Software Engineers. I'm making up the title but the work is real i.e. Software Engineers improving or extending the Cloud Native ecosystem.

This role predominantly exists with vendors with a vested interest in Kubernetes like Cloud vendors or Services companies. Google, for example, has a whole team dedicated to maintaining the Cluster, Horizontal & Vertical Pod Autoscalers.

Pros

Focus on Software Engineering Skills
Potential to learn the internals of various technologies e.g. Prometheus, Docker, Flagger and share with the community
Opportunity/Incentive to represent customer use cases in the larger Cloud Native Community

Cons

No quick “production” feedback like in SRE
Limited Experience operating a kubernetes cluster
Reletaively fewer oppurtunities in the community, but still lots of them!

Developer Advocate

While vendors might hire Software Engineers to help improve their Kubernetes offerings, they might hire Engineers to better understand the community's needs and help guide the product strategy. That's the Developer Advocate role and there's a growing no. of them in the Cloud Native Community.

If you're not familiar with the role, there's a great FAQ by Dustin Ingram.

Pros

Great opportunity to learn in public
- Work on skills such as giving talks, Videography, Writing Blogposts
- A very visible position, great opportunity to create a public profile
Incentive/Oppurtunity to create significantly useful resources for the community:
- Writing books and etc.
- Creating or maintaining to important open source projects
The chance to explore "what’s possible with Kubernetes”:
- Creating lots of “demo” apps to show what's possible with Kubernetes
- Being "user zero" for new products or features
Great opportunity to be valued as an engineer for strengths in:
- Communication
- Stakeholder management
- Product leadership

Cons

Far fewer oppurtunities than a SRE or SWE roles
Role not well defined and may change significantly between companies
- Evangelism vs. Advocate
- Report to marketing vs. director of engineering
- May require lots of travel
Difficult to distinguish between work and hobby projects
Not clear what the employers' non-compete for the role look like:
- If you write a book, create a youtube channel does the employer need to sign off?
Almost no first hand production experience in the role
Hard to measure your impact:
- makes it hard to know what's worth working on
- makes it hard to prove that you're doing a good job

Parting Words

All three roles seem to have a lot to offer and like flavours of an ice cream, they can be mixed and matched. Software Engineers give talks and Site Reliability Engineers write plenty of software. Folks might also be able to move between these roles easily enough.

I hope others find this list as helpful as I did. If you feel like I've missed an important aspect of these or would like to share other kinds of engineering roles you've seen in the Cloud Native community, I'd love to hear from you!

Making the ValidatingWebhook Testable

Muaaz Saleem — Sun, 17 Jan 2021 17:23:46 +0000

In my last post we saw that admitting objects in a ValidatigAdmissionWebhook can be expressed as a pure function i.e.,

func Admit(req *AdmissionRequest) *AdmissionResponse

This time, I'd like to argue that abstracting the Admit func into an Admitter interface can help us make the webhook code more testable.

Testing the `Admit` func

A little aside before getting to the Admitter interface, an additional benefit of wrapping our admission logic in an Admit func, this way, is that it makes it really easy to write a test for it.

func TestAdmit(t *testing.T) {
    // setup req and expected resp
    admResp := Admit(req)
    assert.Equal(t, expectedAdmResp, admResp)
}

The `Admitter` Interface

So our admission logic can now be tesed but what about the HTTP machinery, there's some hairy encoding/decoding logic in our Handler func, so being able to test would be really comforting. That's where the Admitter interface comes in!

Here's the Admit func again:

func Admit(req *AdmissionRequest) *AdmissionResponse

And here's what the Admitter interface could look like:

type Admitter interface {
    // Name() is helpful for logging and metrics
    Name() string
    // Admit() is where the pure admission logic lives
    Admit(req *admissionsv1.AdmissionRequest) (*admissionsv1.AdmissionResponse, error)
}

Our Handler func is going to change just as well to accomodate. The Handler func originally:

func Handler() {
    return func(){
        // read req
        validate()
        // write resp
    }
}

And we call it like:

handler.Handle("/deployments", admission.Handler())

The Handler func now:

func Handler(admitter Admitter) {
    return func(){
        // read req
        admitter.Admit()
        // write resp
    }
}

Calling the Handler func would now look like:

// DeploymentAdmitter implements the Admitter interface
depAdmitter := admission.DeploymentAdmitter{}
handler.Handle("/deployments", admission.Handler(depAdmitter))

Implementing the Admitter Interface

As the last section hinted, each resource that the webhook must validate can just implement the Admitter interface. For example, here's the DeploymentAdmitter:

type DeploymentAdmitter struct {
}

func (d DeploymentAdmitter) Name() string {
    return "deployments"
}

func (d DeploymentAdmitter) Admit(req *admissionsv1.AdmissionRequest) (*admissionsv1.AdmissionResponse, error) {
   // deny if `application` label is missing 
}

And guess what, we can just implement the Admitter interface in our tests to test the Handler. So cool, right!

Here's an example testAdmitter

type testAdmitter struct {
}

func (t TestAdmitter) Name() string {
    return "testAdmitter"
}

func (t testAdmitter) Admit(req *admissionsv1.AdmissionRequest) (*admissionsv1.AdmissionResponse, error) {
   // allow/deny objects depending on the test 
}

Testing the `Handler`

At last, with the actual validation logic abstracted away, testing the Handler is really just as simple as calling the handler with a test request and the testAdmitter:

func TestHandler(t *testing.T) {
    // setup AdmissionReview.Request and expected AdmissionReview.Response objects
    adm = testAdmitter{}
    Handler(req, resp)
    // Test that 
    // 1. when an object should be allowed Handler allows it
    // 2. when an object should be denied Handler denys it
    assert.Equal(t, expectedResp, resp)
}

Thanks to the Admitter interface, we can now test parsing the admission requests and validating them!

Further Resources

This series is based on my experience adding a ValidatingAdmissionWebhook to Skipper, modern HTTP proxy.

Note: While the series uses validating Deployments as an example, the webhook in Skipper validates RouteGroups a Custom Resource used in Skipper. Hope that's not too confusing!

You can find the complete source code for the ValidatigAdmissionWebhook here
Specifically, the tests can be found in admission_test.go

Validating Admission Requests in a Validating Admission Webhook

Muaaz Saleem — Sun, 17 Jan 2021 17:19:14 +0000

Last time, I talked about Parsing Admission Requests in a Validating Admission Webhook. This time I wanna talk about actually validating the admission requests i.e allowing or rejecting new or updated resources.

We learned how to parse a Validation Request in an AdmissionReview.Request Object and serialize (encode) a Validation Response from an AdmissionReview.Response Object.

Now that we know how to parse the HTTP requests, a validating admission webhook is just a go func of the signature:

func validate(req *AdmissionRequest) *AdmissionResponse

Another way to phrase validating an Object is "Admitting an Object". In which case the signature above becomes:

func Admit(req *AdmissionRequest) *AdmissionResponse

Parsing the Object

The first thing to do in the Admit func is to parse the Object we're hoping to validate:

err := json.Unmarshal(req.Object.Raw, &admittedObj)
if err != nil {
    // ...
}

In the example above, I am reading the RouteGroup from AdmissionReview.Request.Object.Raw. Note that you could also deserialize the object from AdmissionReview.Request.Object, I hope to cover that in some other post.

Validating the Object

This is where we finally decide whether an Object is good enough to allow or not. The specific logic will depend on what you want to achieve but as an example suppose our admittedObj above was a deployment and we only wanted to allow it if it had an application label.

The validation would look something like:

err = checkAppLabel(&deployment.Spec.Template)
if err != nil {
    // ..
}

Here's checkAppLabel:

import (
    corev1 "k8s.io/api/core/v1"
)

var (
    errNoApplicationLabel = errors.New(
        "missing application label in  pod template",
    )
)

func checkAppLabel(podSpec *corev1.PodTemplateSpec)
    application, ok := podSpec.Labels["application"]
    if !ok {
        return errNoApplicationLabel
    }

    return registry.CheckApplication(application)
}

So cool, right!

Rejecting an Object

Rejecting an Object is just a matter of filling out the AdmissionResponse object with Allowed: false:

if err != nil {
    emsg := fmt.Sprintf("the object was no good, cuz: %v", err)
    log.Error(emsg)
    return &admissionsv1.AdmissionResponse{
        // The UID is needed to identify which request 
        // you are responsing to
        UID:     req.UID,
        Allowed: false,
        Result: &metav1.Status{
        // The Message is helpful to the users to figure out 
        // what's they did wrong!
            Message: emsg,
        },
    }, nil
}

Allowing an Object

At this point, allowing in object is pretty simple:

    return &admissionsv1.AdmissionResponse{
        UID:     req.UID,
        Allowed: true,
    }, nil

That's all! You can find a complete example of an Admit func from the Validating Admission Webhook in Skipper. I've also included an abridged version at the end of this blog post.

Admission Webhooks are the Swiss Army Knife of a Kubernetes Operator. Once you know how to write one, they can be an easy way to programmatically enforce internal policies cluster wide.

Further Resources

If you'd like to learn more on the topic:

Banzi Cloud has a great guide on admission controllers.
There's also the Dynamic Admission Control guide in the Kubernetes docs.
You can find the definitions of the AdmissionReview object in k8s.io/api repository.

I also like to look at the Kubernetes API reference. You can find the ValidatingAdmissionConfiguration spec there.

Reference

An example Admit func from the Validating Admission Webhook in Skipper for RouteGroups:

func Admit(req *admissionsv1.AdmissionRequest) (*admissionsv1.AdmissionResponse, error) {
    rgItem := definitions.RouteGroupItem{}
    err := json.Unmarshal(req.Object.Raw, &rgItem)
    if err != nil {
        emsg := fmt.Sprintf("could not parse RouteGroup, %v", err)
        log.Error(emsg)
        return &admissionsv1.AdmissionResponse{
            UID:     req.UID,
            Allowed: false,
            Result: &metav1.Status{
                Message: emsg,
            },
        }, nil
    }

    err = definitions.ValidateRouteGroup(&rgItem)
    if err != nil {
        emsg := fmt.Sprintf("could not validate RouteGroup, %v", err)
        log.Error(emsg)
        return &admissionsv1.AdmissionResponse{
            UID:     req.UID,
            Allowed: false,
            Result: &metav1.Status{
                Message: emsg,
            },
        }, nil
    }

    return &admissionsv1.AdmissionResponse{
        UID:     req.UID,
        Allowed: true,
    }, nil
}

Parsing Admission Requests in a Validating Admission Webhook

Muaaz Saleem — Sun, 17 Jan 2021 16:27:09 +0000

Last time I introduced the 2 main parts of a Validating Admission Webhook. The Webserver and the ValidatingAdmissionWebhookConfiguration.

This time I'll expand a bit on the Webserver. Specifically, parsing Admission Requests in a Validating Admission Webhook.

The HTTP Handler

The webserver just needs a POST endpoint, where the API Server can post Admission Requests to.

handler.Handle("/routegroups", admission.Handler())

admission.Handler is just a go func

func Handler() http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        // ...
    }
}

Having a function that returns a http.HandlerFunc is helpful because this way I can also parameterize admission.Handler for each resource I want to validate e.g:

handler.Handle("/deployments", admission.Handler("deployments"))

Parsing an Admission Request

Admission requests are just HTTP Post requests with a JSON body of the form:

{
  "apiVersion": "admission.k8s.io/v1",
  "kind": "AdmissionReview",
  "request": {
    // ...
  }
}

Therefore, we can just use json.Unmarshal to decode them:

import (
    "encoding/json"
    admissionsv1 "k8s.io/api/admission/v1"
)
// ...
review := admissionsv1.AdmissionReview{}
err = json.Unmarshal(body, &review)
if err != nil {
    log.Errorf("Failed to parse request: %v", err)
    w.WriteHeader(http.StatusBadRequest)
    return
}

Note how I am just using the upstream AdmissionReview type from k8s.io/api/admission/v1 here. You can find other Kubernetes types in the k8s.io/api repo as well.

Responding to an Admission Request i.e encoding `Admission.Response`

Encoding is just a json.Marshal as well.

import (
// ...
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

    resp, err := json.Marshal(admissionsv1.AdmissionReview{
        TypeMeta: metav1.TypeMeta{
            Kind:       "AdmissionReview",
            APIVersion: "admission.k8s.io/v1",
        },
        Response: response,
    })

There are a couple important things to keep in mind here.

The response UID must match the request

response = &admissionsv1.AdmissionResponse{
    UID:     &review.UID,
// ...
}

Otherwise, the API Server won't know which admission request my response belongs to.

The TypeMeta fields must be present in the response

admissionsv1.AdmissionReview{
        TypeMeta: metav1.TypeMeta{
            Kind:       "AdmissionReview",
            APIVersion: "admission.k8s.io/v1",
        },
        Response: response,
    })

I learnt this the hard way. If the Kind and APIVersion aren't specified for v1 resources i.e v1.AdmissionReview, the Admission requests will just fail.

Util Funcs

I found it useful to have writeResponse and errorResponse functions.

Writing Response

import (
// ...
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

func writeResponse(
    writer http.ResponseWriter, 
    response *admissionsv1.AdmissionResponse,
    ) {
    resp, err := json.Marshal(admissionsv1.AdmissionReview{
        TypeMeta: metav1.TypeMeta{
            Kind:       "AdmissionReview",
            APIVersion: "admission.k8s.io/v1",
        },
        Response: response,
    })
    if err != nil {
        log.Errorf("failed to serialize response: %v", err)
        writer.WriteHeader(http.StatusInternalServerError)
        return
    }
    if _, err := writer.Write(resp); err != nil {
        log.Errorf("failed to write response: %v", err)
    }
}

Writing errors

import (
// ...
    "k8s.io/apimachinery/pkg/types"
)

func errorResponse(uid types.UID, err error) *admissionsv1.AdmissionResponse {
    return &admissionsv1.AdmissionResponse{
        Allowed: false,
        UID:     uid,
        Result: &metav1.Status{
            Message: err.Error(),
        },
    }
}

When I started writing a validating admission webhook for the first
time, I was very confused about the various moving parts. Hopefully this post shows that in essence, a validating admission webhook is just a go webserver.

Further Resources

You can find an abridged version of admission.Handler under "Reference". For a complete example of a validating admission webhook being used in production, checkout the validating admission webhook in Skipper, our HTTP reverse proxy.

Thanks to my teammates Arpad & Sandor for helping me contribute a validating admission webhook to Skipper!

Reference

The following is just an abridged version of admission.go from the validating admission webhook in Skipper.

package admission

import (
    "encoding/json"
    "fmt"
    "io/ioutil"
    "net/http"
    "time"

    log "github.com/sirupsen/logrus"
    "github.com/zalando/skipper/dataclients/kubernetes/definitions"
    admissionsv1 "k8s.io/api/admission/v1"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/apimachinery/pkg/types"
)

func Handler(admitter Admitter) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        admitterName := admitter.Name()

        if r.Method != "POST" || r.Header.Get("Content-Type") != "application/json" {
            w.WriteHeader(http.StatusMethodNotAllowed)
            return
        }

        body, err := ioutil.ReadAll(r.Body)
        if err != nil {
            log.Errorf("Failed to read request: %v", err)
            w.WriteHeader(http.StatusInternalServerError)
            return
        }
        log.Debugln("request received", r.Method, r.URL.Path, r.Header, string(body))

        review := admissionsv1.AdmissionReview{}
        err = json.Unmarshal(body, &review)
        if err != nil {
            log.Errorf("Failed to parse request: %v", err)
            w.WriteHeader(http.StatusBadRequest)
            return
        }

        operationInfo := fmt.Sprintf(
            "%s %s %s/%s",
            review.Request.Operation,
            review.Request.Kind,
            review.Request.Namespace,
            extractName(review.Request),
        )

        gvr := review.Request.Resource
        group := gvr.Group
        if group == "" {
            group = "zalando.org"
        }

        start := time.Now()
        defer admissionDuration.With(labelValues).
            Observe(float64(time.Since(start)) / float64(time.Second))

        admResp, err := admitter.Admit(review.Request)
        if err != nil {
            log.Errorf("Rejected %s: %v", operationInfo, err)
            writeResponse(w, errorResponse(review.Request.UID, err))
            return
        }

        log.Debugf("Allowed %s", operationInfo)
        writeResponse(w, admResp)
    }
}

func writeResponse(writer http.ResponseWriter, response *admissionsv1.AdmissionResponse) {
    resp, err := json.Marshal(admissionsv1.AdmissionReview{
        TypeMeta: metav1.TypeMeta{
            Kind:       "AdmissionReview",
            APIVersion: "admission.k8s.io/v1",
        },
        Response: response,
    })
    if err != nil {
        log.Errorf("failed to serialize response: %v", err)
        writer.WriteHeader(http.StatusInternalServerError)
        return
    }
    if _, err := writer.Write(resp); err != nil {
        log.Errorf("failed to write response: %v", err)
    }
}

func errorResponse(uid types.UID, err error) *admissionsv1.AdmissionResponse {
    return &admissionsv1.AdmissionResponse{
        Allowed: false,
        UID:     uid,
        Result: &metav1.Status{
            Message: err.Error(),
        },
    }
}

func extractName(request *admissionsv1.AdmissionRequest) string {
    if request.Name != "" {
        return request.Name
    }

    obj := metav1.PartialObjectMetadata{}
    if err := json.Unmarshal(request.Object.Raw, &obj); err != nil {
        log.Warnf("failed to parse object: %v", err)
        return "<unknown>"
    }

    if obj.Name != "" {
        return obj.Name
    }
    if obj.GenerateName != "" {
        return obj.GenerateName + "<generated>"
    }
    return "<unknown>"
}

The 2 main parts of a Kubernetes Validating Admission Webhook

Muaaz Saleem — Fri, 06 Nov 2020 11:32:51 +0000

Validating Admission Webhooks

Validating Admission Webhooks can be used to intercept requests to the API Server and validate new and updating resources. For example, you could have a Validating Admission Webhook that keeps new Deployment objects from being created if they're missing an application label. You could also keep certain resources from being updated.

A Validating Admission Webhook has 2 main parts:
- A Webhook Server
- A ValidatingAdmissionWebhookConfigurationresource

Note: The code examples and a couple passages in this post are borrowed from the Dynamic Admission Control reference. Shared under the Creative Commons Attribution 4.0 International Lisence.

The Webhook Server

The webhook server is just a webserver that handles "validation" requests from the API server.

These requests are regarding specific resource types and object changes. For example, creation of a Deployment resource. This is specified in the ValidatingAdmissionWebhook for the admission webhook.

The webserver communicates with the API Server over HTTPs (TLS) and therefore requires a TLS cert and key pair.

Requests

Webhook Servers a.k.a webhooks, are sent a POST request, with Content-Type: application/json, with an AdmissionReview.Request API object in the admission.k8s.io API group serialized to JSON as the body.

This example shows the abridged data contained in an AdmissionReview object for a request to update the scale subresource of an apps/v1 Deployment:

{
  "apiVersion": "admission.k8s.io/v1",
  "kind": "AdmissionReview",
  "request": {
    # Random uid uniquely identifying this admission call
    "uid": "705ab4f5-6393-11e8-b7cc-42010a800002",
    # Name of the resource being modified
    "name": "my-deployment",
    # Namespace of the resource being modified, if the resource is namespaced (or is a Namespace object)
    "namespace": "my-namespace",
    # operation can be CREATE, UPDATE, DELETE, or CONNECT
    "operation": "UPDATE",
    # object is the new object being admitted.
    # It is null for DELETE operations.
    "object": {"apiVersion":"autoscaling/v1","kind":"Scale",...},
    # oldObject is the existing object.
    # It is null for CREATE and CONNECT operations.
    "oldObject": {"apiVersion":"autoscaling/v1","kind":"Scale",...},
    # dryRun indicates the API request is running in dry run mode and will not be persisted.
    # Webhooks with side effects should avoid actuating those side effects when dryRun is true.
    # See http://k8s.io/docs/reference/using-api/api-concepts/#make-a-dry-run-request for more details.
  }
}

Response

The webhook server responds with a 200 HTTP status code, Content-Type: application/json, and a body containing an AdmissionReview.Response object (in the same version they were sent), with the response stanza populated, serialized to JSON. At a minimum, the response stanza must contain the following fields:

uid, copied from the request.uid sent to the webhook
allowed, either set to true or false

Example of a minimal response from a webhook to allow a request:

{
  "apiVersion": "admission.k8s.io/v1",
  "kind": "AdmissionReview",
  "response": {
    "uid": "<value from request.uid>",
    "allowed": true
  }
}

Example of a response to forbid a request, customizing the HTTP status code and message:

{

  "apiVersion": "admission.k8s.io/v1",

  "kind": "AdmissionReview",

  "response": {

    "uid": "<value from request.uid>",

    "allowed": false,

    "status": {

      "code": 403,

      "message": "You cannot do this because it is Tuesday and your name starts with A"

    }

  }

}

ValidatingWebhookConfiguration

The ValidatingAdmissionWebhookConfiguration specifies what object type and request methods should the webserver be called on. The Configuration resource also specifies the TLS caBundle the API server should use to connect with the webserver.

An example ValidatingWebhookConfiguration:

apiVersion: admissionregistration.k8s.io/v1

kind: ValidatingWebhookConfiguration

metadata:

  name: "pod-policy.example.com"

webhooks:


name: "pod-policy.example.com"
rules:


apiGroups:   [""]
apiVersions: ["v1"]
operations:  ["CREATE"]
resources:   ["pods"]
scope:       "Namespaced"
clientConfig:
service:
namespace: "example-namespace"
name: "example-service"
caBundle: "Ci0tLS0tQk...<caBundle is a PEM encoded CA bundle which will be used to validate the webhook's server certificate.>...tLS0K"
admissionReviewVersions: ["v1", "v1beta1"]
sideEffects: None
timeoutSeconds: 5

Further Resources

To learn more about validating admission webhooks and dynamic admission control in general, see the Dynamic Admission Control reference from the Kubernetes docs.
You can find the definitions of the AdmissionReview object in k8s.io/api repository.
I also like to look at the Kubernetes API reference. You can find the ValidatingAdmissionConfiguration spec there.

Forem: Muaaz Saleem

3 Principles I wanna apply at work in 2021

Accept the Chaos

Embrace Spontaneity

Reduce Work In Progress

Excited to look back

Pros and Cons of Using client-go in Kubernetes controllers

Pros of using client-go

Helpful Utilities

Implementations for all kubernetes API Objects

Powerful abstractions for common use casses

Cons of using client-go

Cognitive Overload

Scarce Docs

Not the most idiomatic go-code

Further Resources

Creating a Go Kubernetes client with client-go

A client-go "Hello, World!"

Creating Client Config

Creating out of cluster config

Creating in cluster config

Putting it all together

Testing with Fake Client

Managing package dependencies

Further Resources

5 Time saving things about client-go, I didn't know!

Creating a Test HTTP Server in Go

Usage

A more complete example

pingService()

Testing pingService()

Further Resources:

The Three Main Flavors of Kubernetes related Engineering Roles

Site Reliability Engineer

Pros

Cons

Software Engineer ( Kubernetes )

Pros

Cons

Developer Advocate

Pros

Cons

Parting Words

Making the ValidatingWebhook Testable

Testing the Admit func

The Admitter Interface

Implementing the Admitter Interface

Testing the Handler

Further Resources

Validating Admission Requests in a Validating Admission Webhook

Parsing the Object

Validating the Object

Rejecting an Object

Allowing an Object

Further Resources

Reference

Parsing Admission Requests in a Validating Admission Webhook

The HTTP Handler

Parsing an Admission Request

Responding to an Admission Request i.e encoding Admission.Response

Util Funcs

Further Resources

Reference

The 2 main parts of a Kubernetes Validating Admission Webhook

Validating Admission Webhooks

The Webhook Server

Requests

Response

ValidatingWebhookConfiguration

Further Resources

A `client-go` "Hello, World!"

`pingService()`

Testing `pingService()`

Testing the `Admit` func

The `Admitter` Interface

Testing the `Handler`

Responding to an Admission Request i.e encoding `Admission.Response`