Forem: Ryan, Siu Long Wa

Google Cloud Build Visualizer

Ryan, Siu Long Wa — Sun, 21 Nov 2021 08:53:21 +0000

Google Cloud Build (GCB) is a building system provided by GCP. Considering all the competitors like GitLab CI, CircleCI and Travis CI, GCB is definitely not a common option for building CI CD pipeline.

In fact, I have tried to test GCB in my work at the early stage of adoption in GCP. The pros of GCB is definitely a full integration with GCP's resources. If you are using GCP and have no preference at the CI pipeline platform, I highly suggest you to try it out.

Circling back to the GCB's pipeline, all the configurations are written in JSON or YAML format. Here is an example for the GCB configuration.

steps:

  - id: 'compile web app'
    name: 'gcr.io/cloud-builders/npm'
    dir: 'web'
    args: ['install']

  - id: 'web app unit tests'
    name: 'gcr.io/cloud-builders/npm'
    dir: 'web'
    args: ['test']
    waitFor: ['compile web app']

  - id: 'build web'
    name: 'gcr.io/cloud-builders/docker'
    args: [
      'build',
      '--tag=web', # use local registry for compatibility with local builds
      '--tag=gcr.io/$PROJECT_ID/web',
      '--cache-from=gcr.io/$PROJECT_ID/web:latest',
      'web/.',
    ]
    waitFor: ['web app unit tests']

  - id: 'build db'
    name: 'gcr.io/cloud-builders/docker'
    args: [
      'build',
      '--tag=mysql', # use local registry for compatibility with local builds
      '--tag=gcr.io/$PROJECT_ID/mysql',
      '--cache-from', 'gcr.io/$PROJECT_ID/mysql:latest',
      'mysql/.',
    ]
    waitFor: ['-'] # start immediately

  - id: 'compose up'
    name: 'gcr.io/$PROJECT_ID/docker-compose:latest'
    entrypoint: '/bin/bash'
    args:
      - '-c'
      - |
        docker-compose up -d
    env:
      - 'PROJECT_ID=$PROJECT_ID'
    waitFor: ['build web','build db']

  - id: 'integration tests'
    name: 'gcr.io/$PROJECT_ID/docker-compose:latest'
    entrypoint: '/bin/bash'
    args:
      - '-c'
      - |
        ### -r = retries; -i = interval; -k = keyword to search for ###
        ./test/test-connection.sh -r 20 -i 3 -u http://web:3000
        ./test/test-content.sh -r 20 -i 3 -u http://web:3000 -k 'Chocolate Chip'
    waitFor: ['compose up']

images:
  - gcr.io/$PROJECT_ID/web
  - gcr.io/$PROJECT_ID/mysql

Look intuitive? Yeah! It looks intuitive. It only has the following rules in the pipeline sequence.

No parallel by default
You can specify the waitFor to specify the dependencies. Each dependencies will be executed in parallel.
You can use special character - to specify this step for job without dependency to run in parallel.

So, what is the problem with GCS? The issue here is that the UI itself does not provide you with a clear view on how the steps are depended on each other. Here is an example.

Here, you can see the steps are showed in sequential manner. You cannot observe if this build has been executed in parallel.

Therefore, I have tried to build a simple CLI to make sure we can understand how the steps are depended.

RyanSiu1995 / gcb-visualizer

Cloudbuild pipeline visualizer with graphviz

Google Cloud Build Pipeline Visualizer

For the current version of Google cloud build, it supports the async process with the variable waitFor. With the growth of complexity of your pipeline, it will be hard to maintain the async flow. Unlike Jenkins and CircleCI, there is no visualizer for your pipeline. This application aims at visualize the pipeline and help the developers to debug their cloud build.

Current features

YAML format cloud build definition digestion
Temporary graph rendering
Save graph as dot, png, jpg or jpeg
JSON format support

Rule of cloud build async process

From the Google docs, there are a few rules for the async process.

If no values are provided for waitFor, the build step waits for all prior build steps in the build request to complete successfully before running.
A step is dependent on every id in its waitFor and will not launch until each…

View on GitHub

In this project, I built the CLI with cobra and graphviz. The logic is so simple that it will intake the JSON or YAML file. Then, it parses the configuration and turn the pipeline into a graph. Finally, the graph can be visualized by graphviz.

You can get the CLI easily with the following command.

go get -u github.com/ryansiu1995/gcb-visualizer

Go to the link and check it out if you are using GCB in your company or personal project. It definitely helps you to understand how you have configured your pipeline in GCB.

Please feel free to open an issue if you have configured out any features you want in the GCB Visualizer!

Quick walkthrough in CKAD, CKA and CKS

Ryan, Siu Long Wa — Wed, 24 Mar 2021 08:31:48 +0000

Certified Kubernetes Application Developer and Certified Kubernetes Administrator are the two certifications for most of the Kubernetes users to showcase their understanding in the Kubernetes ecosystem.

Certified Kubernetes Security Specialist is the certification released last year to train and showcase the candidates on Kubernetes system hardening. It is definitely a new step for the community getting DevSecOps.

In the past few months, I have finished all the examinations. Here is my review and analysis for these examinations for not only examinees but also the recruiter.

Certified Kubernetes Application Developer (CKAD)

Certified Kubernetes Application Developer is my first examination among the series. For the actual examination details, please refer to the actual documentation from CNCF.

CKAD, as its name, aims to train and examine the basic Kubernetes operation. The examinee will need to demonstrate their understanding in application deployment under Kubernetes.

Most of the questions inside CKAD can be generalized into the concepts or best practice under Kubernetes ecosystem. Therefore, if you understand how to architect the apps inside Kubernetes, this exam will be easy for you.

Example - Sidecar pattern

If you need to deploy an application which will write the logs to file instead of the stdout, how could you stream the logs to stdout without changing the actual configuration? This will require the sidecar pattern under Kubernetes. Here is the example from the Kubernetes doc.

apiVersion: v1
kind: Pod
metadata:
  name: counter
spec:
  containers:
  - name: count
    image: busybox
    args:
    - /bin/sh
    - -c
    - >
      i=0;
      while true;
      do
        echo "$i: $(date)" >> /var/log/1.log;
        echo "$(date) INFO $i" >> /var/log/2.log;
        i=$((i+1));
        sleep 1;
      done      
    volumeMounts:
    - name: varlog
      mountPath: /var/log
  - name: count-log-1
    image: busybox
    args: [/bin/sh, -c, 'tail -n+1 -f /var/log/1.log']
    volumeMounts:
    - name: varlog
      mountPath: /var/log
  volumes:
  - name: varlog
    emptyDir: {}

Under this example, a couple of concepts can be extracted.

The first concept, of course, is the sidecar pattern. The sidecar pattern will be important when the software engineers architect their application under Kubernetes. It is very important that the software engineers make use of the sidecar pattern to achieve the best practice, single process per container.

The second concept here is the temporarily storage. It is common that an empty directory is created within a pod and shares the files across different containers.

Example - Usage in the Deployment, DaemonSet and StatefulSet

CKAD examinees in general are required to demonstrate their understanding between Deployment, DaemonSet and StatefulSet. Although they can be used for deploying your applications, they are served for different purpose based on the different usage.

If you are asked to deploy FluentD, a log aggregator, to stream all the logs to your own logging system, what type of ReplicaSet you will use? You can simply inject the sidecar to each pod but that is hard to manage. The better way on this will be the DaemonSet. Deploying the containers interact with each node can help you to a cluster-wide log streaming in this case.

Who should take this examination?

CKAD is quite different from the CKA and CKS. It aims to test the users on the basic architecture and design pattern under Kubernetes. If you are the following group of engineers, I suggests that you can give a try on this certification.

Beginner in Kubernetes
Software engineers who use Kubernetes for your production environment
DevOps/Site Reliability Engineers
System Engineers
etc...

After the examination, you can expect that you are able to use Kubernetes to architect and design your application.

Certified Kubernetes Administrator (CKA)

If CKAD is a beginner under Kubernetes certification series, CKA will be the next level. It will cover parts of the CKAD knowledge, for example, the application development. At the same time, CKA examinees will start to interact with the actual Kubernetes implementation.

You need to understand how the Kubernetes is built behind the scene. Unlike CKAD, you need to configure not only the highest level of objects inside Kubernetes, like Pod, Deployment but also the actual key components inside the cluster.

In the community, there are a couple of ways to bootstrap the cluster. In the examination, it only focuses on the kubeadm method. The operation, like node joining, upgrading, etc, will be asked. It is quite similar to the system administrator but it focuses on the maintenance under Kubernetes.

Example - `kubelet`, `kube-apiserver` and `etcd` configuration

Under kubeadm implementation, you can easily configure the args parsing into the kubelet, kube-apiserver and etcd. But unlike configuring the Kubernetes objects, you cannot use kubectl to interact with these components directly. You will need to understand how the kubeadm bootstraps a node. In general, kubeadm generates a set of configuration files under a particular path.

Example - CNI and CRI

Container Network Interface (CNI) and Container Runtime Interface (CRI) are two important interfaces under Kubernetes. This allows us to change the implementation of the container networking and runtime easily.

But, you may ask why you will need to change them. Different CNIs' and CRIs' implementation actually provides different features. According to your business need, you may need to choose the best CNI and CRI. Beyond choosing the correct one, you will need to understand how to configure them within the clusters.

Who should take this examination?

CKA starts to interact with kubelet, kube-apiserver and etcd directly. You will need to understand how the Kubernetes works behind the scene and how to maintain it. Here are the candidates that may benefit from passing the examination in my mind.

Advanced Kubernetes users, like operator maintainer
DevOps/Site Reliability Engineers
System Engineers
Security Engineers

For Advanced users, they want to interact with the kube-apiserver and etcd to develop their own controllers or operators, the knowledge from CKA is quite helpful to you.

For the last one, it is important for the security engineers to understand how the Kubernetes is bootstrapped. Kubernetes is more and more prevalent in recent years. When you works for a company with Kubernetes orchestration, you definitely need to explore Kubernetes from the system level instead of simply the usage level.

Certified Kubernetes Security Specialist (CKS)

CKS is the last certification. You need to pass the CKA in order to get this certified. Therefore, you can know that the difficulty of CKS will not be lower than CKA.

CKS aims to examine the candidates from more advanced usage. Under CKAD and CKS, you only focuses on the built-in APIs from Kubernetes. You can open the API cookbook under Kubernetes.io to finish most of the tasks in the examination. But, for CKS, you start to integrate your cluster and applications with the community-driven application.

Example - Annotation-based API

The most important part here is how to extend the Kubernetes features with annotation. Security is simply the smokescreen. In Kubernetes ecosystem, if there are some features not coming with the official APIs, it is pretty common that the APIs are placed in the annotation.

For example, one important third party application under Kubernetes ecosystem is cert manager. When you configure an ingress with cert manager, you can put the annotation like the following example from official doc to tell the cert manager to generate TLS cert for you.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    # add an annotation indicating the issuer to use.
    cert-manager.io/cluster-issuer: nameOfClusterIssuer
  name: myIngress
  namespace: myIngress
spec:
  rules:
  - host: example.com
    http:
      paths:
      - backend:
          serviceName: myservice
          servicePort: 80
        path: /
  tls: # < placing a host in the TLS config will indicate a certificate should be created
  - hosts:
    - example.com
    secretName: myingress-cert # < cert-manager will store the created certificate in this secret.

Example - API server and `Kubelet` Hardening

In CKA, you need to configure the API server and kubelet with the basic function. No matter what configuration you have in the API server and kubelet, it is more than enough for that evaluation.

In CKS, you will need to configure the cluster with the best practice. Kubernetes is designed to fit in a lot of different situations. Therefore, some of the settings may contain the attack surface if you enable them. If you put the Kubernetes into production, you need to fine tune your configuration in order to prevent the cluster from hacking.

Who should take this examination?

CKS is the latest certification released by CNCF. It actually brings the Kubernetes certification series into more production level. In CKAD and CKA, you can assume the certified can use the Kubernetes. But that is the best for the production usage.

Under the production usage, you definitely need to consider the security no matter which industry you are in. CKS provides a way to teach the candidates how to harden the Kubernetes system. Here are the groups I recommend them to take this.

DevOps/Site Reliability Engineers
System Engineers
Security Engineers

A side track message - Please be reminded that security is not just the responsibility of security engineers.

Conclusion

To be honest, as a Kubernetes engineer, CNCF has tried their best in the design of these three certification. Unlike other certifications, you will need to have the hands-on operations during the examination.

One more suggestion from me is that you will need to be super super familiar in living in shell. If you used the dashboard or UI for your Kubernetes journey, it is time to change your habit!

Disclaimer

I have tried my best not to disclose any kind of examination questions within my sharing. Please leave the comments or email me at findme@ryansiulw.com if there are any improper contents. I will remove that part or simply delete this sharing directly.

Dev Blog - Celery Operator - Part 1 - Initial Design

Ryan, Siu Long Wa — Sat, 05 Sep 2020 13:27:51 +0000

This is the first part of my development blog in celery operator. Here is the disclaimer. This operator is purely my personal learning product. It does not go through the official CEP process in celery community. Please refer to the official one if you seek for an operator with LTS.

Background

With the heavy usage of operators in my career, the abstraction in the Kubernetes layer is so awesome that engineers could easily monitor and scale their applications with kubectl.

In my current company, in order to separate the loading from web server, some of the tasks are delegated to a task queue application. It is called celery in python. A ton of apps are now supporting celery as their task queuing backend.

The great thing of celery is similar to other task queuing service. It provides multiple queues for different requirements from the task. It also provides us with a cronjob service to run tasks periodically. This provides a super handy way to create both on-demand and corn-based task execution to a web server.

Issues spotted

In the Kubernetes world, the biggest advantages Kubernetes has are scaling and observability.

For example, it is common to implement the horizontal pod autoscaler(HPA) with different affinity and selector rules. High availability and high redundancy becomes a super easy task to accomplish in this realm.

Object status is also a powerful utility when we deal with the custom resource definition. It provides us with a handy way to list out the status of apps itself. If you have deal with operator before, you will know a good operator design could give you a full overview in kubectl interaction.

Let’s circle back to the celery itself. My experience in the deployment of celery onto Kubernetes is so bad that the app itself could not fully utilize the modern infra design in the Kubernetes.

For example, celery requires a number of resources in order to provide the full functioning app stack. Broker, result backend, worker and schedule are required. In my current company, the helm chart to deploy celery stack is pretty complex. This handicaps our maintainability of the charts.

Observability is also a serious issue in celery. Celery does not come with tracing or monitoring feature. We need to deploy an additional app called flower to show up the status of celery. Unfortunately, my colleagues had a really terrible experience in flower... It consumed a myriad of resources in his laptop... Anyway, in order to provide more detailed info, we may need to enter the pods and execute the celery command. This introduces a serious security issue because you gives out the permission of exec in the cluster to the engineers. Eventually, this increases the workload in my team drastically when dealing with celery.

Last but not least, the scaling with Kubernetes is so terrible for celery. In order to scale the celery against with the queue size (of course based on queue size but not the built-in CPU or memory way). We need to setup celery exporter and Prometheus. And Prometheus turns the metric into Kubernetes-native metrics format. Umm... When you see this action, you know how complicated it is... But without scaling, we need to pay extra cost in maintaining those potentially unused instances...What a dilemma it is!

Design of operator

In order to resolve the issues spotted, I have decided to move onto operator. Originally, I just want to make use of the official one. But I just realized that the official one is still under POC and it cannot fully support what I want to do in the operator.

Therefore, I have decided to design and develop my own celery operator!

We have a ton of methods to build a Kubernetes operator. As the officially supported celery operator, it takes an approach like a templating method, similar to helm and kustomize. For my celery operator, I want to try something different, operator SDK with kubebuilder!

Originally, I want to create an all-in-one celery CRD. However, after a careful thinking, I felt all-in-one celery CRD is pretty hard to maintain. Imagine that there are over 1000 lines within a reconcile function. This definitely is not a good style in coding.

Therefore, I eventually separated all the controllers and use an all-in-one celery CRD to wrap the rest of CRDs. The resultant structure is like this.

type Celery struct {
  Schedulers []CeleryScheduler
  Broker CeleryBroker
  Workers []CeleryWorker
}

Or using graphviz to visualize the relation.

With this design, the users could simply fit in their own usage. If they want to keep their existing stack with a newly introduced workers, they could simply create the workers CRD. And the code itself could be separated easier.

Under the controller, I originally chose the deployment as the way to control the pod spawning. However, I have foreseen that deployment possibly is not a good way to go if I want to control the pod spawning with custom metrics, like queue size.

Eventually, I change the design from deployment to pod directly. This is a really tough decision because I needed to implement all the CRUD issues on my own... This is another story to go...

For now, the celery operator has already implemented the create, update and delete method. The last step before releasing the first Alpha version is the metric extraction. This will be included in the next article in this series.

Ref:
celery-operator - https://github.com/RyanSiu1995/celery-operator

Will you map Ctrl to CapLock?

Ryan, Siu Long Wa — Tue, 25 Aug 2020 02:52:25 +0000

In my keyboard setting, I always map ctrl key to caplock such that I could press the ctrl key easier. However, this makes a big trouble when someone uses my keyboard lol. I tried to find a keycap that will indicate this change but cannot find any vendors provided this printing service.

How's about you? Do you map the key?

A quick way to prune local branches

Ryan, Siu Long Wa — Tue, 18 Aug 2020 14:44:05 +0000

When I worked with a lot of features at the same time, it was really common to create a bunch of branches locally.

I felt really annoyed when I run git branch locally... It gives me a ton of legacy branches... This is super annoying especially when I use the tab-complete function.

When I searched for the way to clean up my local environment, it is common that the community will suggest git fetch origin -p. But, in fact, this is just a way to clean up the remote branches cached locally, i.e. the branches you see when you run git branch -r.

It seems that the current git does not implement such handy feature. I have quickly created this snippet in GitHub.

#!/bin/bash -e

git fetch origin -p
REMOTE=$(git branch -r | sed "s/origin\///g")
BRANCHES=$(git branch | awk -F ' +' '! /\(no branch\)/ {print $2}')
for branch in $BRANCHES; do
  if [[ -z $(echo $REMOTE | grep -w $branch) ]]; then
    git branch -D $branch
  fi
done

Ref: https://gist.github.com/RyanSiu1995/da44646f747623d9f51ec52724f16066

Once you create a file with this code, run the following command to put it to your PATH.

chmod 777 git-prune-branches
mv git-prune-branches /usr/local/bin

Enjoy!

Kubebuilder Installer

Ryan, Siu Long Wa — Mon, 17 Aug 2020 13:48:23 +0000

Kubebuilder Installer

This is the sugar action that helps me to replace the script inside my operator action development.
It will download the corresponding version of kubebuilder and keep it under /usr/local/kubebuilder.

Submission Category:

Maintainer Must-Haves, Wacky Wildcards (Mainly used for the CI in Kubernetes operator development)

Yaml File or Link to Code

RyanSiu1995 / kubebuilder-action

GitHub Action module to install kubebuilder

Kubebuilder installation docker action

This action provides a sugar syntax to install the kubebuilder in the ubuntu machine. The installation process is based on kubebuilder v2.3.1.

Inputs

`version`

Required The version going to be installed. Default "2.3.1".

Example usage

uses: RyanSiu1995/kubebuilder-action@v1
with:
  version: 2.3.1

License

This software is released under Apache License 2.0. Please refer to LICENSE file for more details.

View on GitHub

Additional Resources / Info

The first version of this action is just to use the child_process inside JS. In the next release, it will change back to the standard libraries in JS.

Interview to GAFA

Ryan, Siu Long Wa — Fri, 10 Jul 2020 16:35:49 +0000

With two years experience in IT and non-CS background, I really want to challenge myself and see how far I can go. Then, I have applied for the big tech companies. Eventually, I got defeated but I know more about my weaknesses.

Coding Test

All positions in the big tech companies must start with coding challenge. The coding challenge in general does not aim to make you feel bad. Instead, you need to present your mindset verbosely and show how well you understand the complexity and data structure.

It is not necessary to run the code but analyze your code well. As a starting point, it is better if you can actually explain your mindset first.

And one thing I have learned. No one cares if you could memorize all the standard libraries. Spend more time on your oral presentation, algorithm design and coding practice instead!

Deep understanding on computer

I lived in a practical world. I learned all the Linux and networking from my work. However, it is really common to test your understanding on the story behind the scene.

For every button, every single step and every command you run, you have to know it from its root. For example, when you talk about TCP, you need to know how it does the three way handshake. But this is just a basic. You need to also memorize and know all the parts inside the pocket and how they actually make the TCP works. Always ask why when you hit an action in your work.

Think about the private cloud

A lot of companies nowadays use public cloud. But when you enter the big tech companies, they use private cloud instead. You need to know how to manage the cluster in the data center, like rack.

And you also need to know how to handle the distributive computing inside the private cloud.

You will hit a lot of problems if you only know how to handle the public cloud, especially for those who only know how to point and click in the UI.

Sum up

I am still too green in the IT industry. I have planed for studying the basic knowledge again and consolidate all the area before taking their challenges again.

My target is to finish my selected MIT open course first!

Forem: Ryan, Siu Long Wa

Google Cloud Build Visualizer

RyanSiu1995 / gcb-visualizer

Cloudbuild pipeline visualizer with graphviz

Google Cloud Build Pipeline Visualizer

Current features

Rule of cloud build async process

Quick walkthrough in CKAD, CKA and CKS

Certified Kubernetes Application Developer (CKAD)

Example - Sidecar pattern

Example - Usage in the Deployment, DaemonSet and StatefulSet

Who should take this examination?

Certified Kubernetes Administrator (CKA)

Example - kubelet, kube-apiserver and etcd configuration

Example - CNI and CRI

Who should take this examination?

Certified Kubernetes Security Specialist (CKS)

Example - Annotation-based API

Example - API server and Kubelet Hardening

Who should take this examination?

Conclusion

Disclaimer

Dev Blog - Celery Operator - Part 1 - Initial Design

Background

Issues spotted

Design of operator

Will you map Ctrl to CapLock?

A quick way to prune local branches

Kubebuilder Installer

Kubebuilder Installer

Submission Category:

Yaml File or Link to Code

RyanSiu1995 / kubebuilder-action

GitHub Action module to install kubebuilder

Kubebuilder installation docker action

Inputs

version

Example usage

License

Additional Resources / Info

Interview to GAFA

Coding Test

Deep understanding on computer

Think about the private cloud

Sum up

Example - `kubelet`, `kube-apiserver` and `etcd` configuration

Example - API server and `Kubelet` Hardening

`version`