Forem: Beingana Jim Junior

Implementing Resource Versioning in Conveyor CI

Beingana Jim Junior — Sat, 09 Aug 2025 23:02:51 +0000

Resource versioning is a key feature in Ci/CD Platforms. It provides developers a way to track how a pipeline has changed over time, ensuring reproducibility, stability, and traceability of builds. Conveyor CI on the other hand, doesn't have this inbuilt into it. This has therefore been a great downside of it, leading developers who might use it miss out on the above mentioned features.

How it is designed in other systems

The design or implementation can vary depending on multiple cases like the purpose, use case, or even the internal design of the system. But at the core, you need to be able to differentiate multiple variations of the same process as it is re-triggered. This means each execution of a CI/CD process has to have a unique Identifier that differentiates it from other executions under that same resource. The most common identifiers or ways in which this differentiation is done include:

Semantic Versioning (SemVer): This is a famous software versioning scheme that follows the MAJOR.MINOR.PATCH eg 1.2.4. It is easy to understand and commonly used in CI/CD systems like GitHub Actions, Gitlab CI and Jenkins.
Commit SHA Pinning: This is a method commonly used in Git based CI/CD systems, whereby they pin the Git Commit to a version and one major advantage of this is it ensures absolute reproducibility.
Custom Revision Generations: In this case a unique string is attached to a resource version. It doesn't have to have any semantic meaning but has to be unique.

Intended Design in Conveyor CI

Conveyor CI being a minimal State Based system that uses a resource driven architecture, it won't base on any external system to implement versioning rather on system state. This means we are free to implement our own versioning scheme. The goal is to use a scheme that is intuitive on the human side and semantically.

With this in mind, we can opt for an incremental integer based versioning scheme. Meaning we just keep incrementing a positive integer starting from zero with each new resource version created.

Technical Implementation

The proposed design can seem like a simple and easy concept to implement, but just like all technical problems, choosing the right implementation might not be as straight forward as it seems. We had to explore different ways to implement this.

A straightforward way might be that each time a resource is updated, a new resource is created with a resource version increased by one and its stored in the database, then a pointer is created so that when a user requests for the current/latest resource version we provide the recent update added.

But this introduces an expected bottle neck. We are assured to blot the database as resources get more revisions. For example, assuming we have 1000 resources and each resource has 100 revisions. That's automatically one hundred thousand records in the database. And say our system becomes big and he has around one million resources each with 100 revisions. The database becomes huge with 100 million records.

So we have to then take into consideration our data store and investigate if it's able to handle this form of incrementation. Conveyor CI uses etcd, a key-value store, it is reliable and highly performant. As we investigated further into the architecture of etcd, we realized that internally etcd uses Multi-Version Concurrency Control (MVCC) which allows reads at specific revisions of a record or key. This means for every update to an etcd key you make, etcd stores the previous version of that record and you can query or read it later on. So technically, rather than having to create a new record each time we update a resource, we can rely on etcd’s internal MVCC to store revisions. Another upside to this is that etcd also uses the incremental revision numbering format so we can also rely on that.

However, the designers of etcd also released that storing revisions, introduces database blot as the records keep increasing in number and as they keep being updated. So they designed a concept of compacting whereby the database will periodically delete old revisions and keep only the latest version of a record. But this now becomes a danger to the data integrity of conveyor ci, meaning we wont be able to access old revisions since they will be permanently deleted. Luckily, this feature of compacting automatically can be turned off and compacting can be done manually by a system administrator. So we can utilize this manual compacting to our benefit and design our own custom compact strategy.

Compact Strategy

To prevent the etcd from becoming extremely huge and also capitalize on the compact feature in etcd. We can introduce a custom compact strategy for systems that have managed to generate an extremely large database and performance is degrading. Our strategy works as follows:

Enable manual etcd compacting
Introduce an External Audit Storage.
Upon compacting, we store a snapshot of the revisions in the External Audit storage system. Then compact etcd
When a user tries to query extremely old revisions, we can then refer and collect them from the external audit system

This ensures that ou etcd datastore remains efficient without Key Explosion and avoids the performance degradation and storage inefficiency of millions of custom historical keys. The external Audit storage system in some cases can also work as a backup system.

The above mentioned implementation is currently the best we have and has actually been proved to be production ready by large systems like kubernetes, and it is what will be patched to Conveyor CI, unless proven otherwise or a better implementation is proposed.

A Complete Guide to etcd: The Distributed Key-Value Store Powering Cloud Infrastructure

Beingana Jim Junior — Fri, 25 Jul 2025 21:41:22 +0000

etcd is a distributed key-value store designed for reliability, high availability, and consistency. It is a Cloud Native project that powers much of the tools utilized in the Cloud Ecosystem today. It is used projects like Kubernetes, CoreOS, OpenShift, Cloud Foundry any many more. It is primarily used to store configuration, state and metadata in Cloud Systems. This article will take you through a complete guide to understanding etcd to the point of being able to integrate it into your project. We shall cover how to install it, setup a cluster, store and query etcd, operating and managing etcd and many more concepts. By the end of this blog article, you will have a solid understanding of etcd.

This article is accompanied with a GitHub repository containing reference code and config files. I would recommend cloning the repository so you can use it to follow along. You can find the code on this repo https://github.com/jim-junior/etcd-guide.git. That being said, this is a long article so let's get started.

Installation

To use etcd you will need to first install it on your system or have an install of it running somewhere. There are multiple ways to install etcd and each method can vary depending on your use-case. This tutorial will try to provide an exhaustive list depending on common use cases.

Installing on a host machine

This is one of the most common ways you will install software if you want to use it on your workspace. When installing etcd, it comes with two important binaries, the etcd server binary and etcdctl CLI tool for interacting with your etcd cluster or instance.

To install these tools you can head over to the Github Realeses page and download the archive for your platform OR

For Linux & MacOS

To install on linux, In the Github repository, I provide a custom installation script that will install the latest version for you. Just clone and run the etcd-install.sh file

You can also install it via Homebrew if you have it installed on your system via brew install etcd

For Windows

To install on WIndows, In the Github repository, I also provide a custom Powershell installation script that will install the latest version for you. Just clone and run the Install-Etcd.ps1 file

Once you have etcd installed, you can verify by running the etcd command:

$ etcd
{"level":"warn","ts":"2025-07-24T16:58:58.804311+0300","caller":"embed/config.go:1209","msg":"Running http and grpc server on single port. This is not recommended for production."}

Installing as a Container

You can also install and run etcd via Docker. The etcd team provides an official container image that can be downloaded from quay.io/coreos/etcd. However, there is another image provided by bitnami and you can pull it from the bitnami/etcd repository.

docker run  -p 4001:4001 -p 2380:2380 -p 2379:2379 --name etcd quay.io/coreos/etcd:v2.3.8
# OR
docker run -it -p 4001:4001 -p 2380:2380 -p 2379:2379 --name etcd bitnami/etcd:latest

If you are using Kubernetes, you can also Install and run it on Kubernetes as a Statefull set, follow the official Tutorial on the etcd documentation

Setting up an etcd cluster

As mentioned earlier, etcd is a distributed KV store. So by nature it can operate in a cluster whereby you have multiple instances running on different nodes. This makes etcd highly fault tolerant. This distributed nature is powered by The Raft Consensus Algorithm and the etcd team actually provides a really good Raft golang implementation. To set up a cluster you need around 3 to 5 instances for high availability and quorum.

Let's go through how you can set up an etcd cluster.

An etcd cluster is composed of multiple instances running and each instances requires an few requirements which include:

Name: A string to identify each instance and it must be unique across instances.
Peer URL: This is a URL used by peer instances to communicate among each other in the cluster. (Usually listens on Port 2380)
Client URL: This is a URL used by external services to interact with your etcd instance. (Usually listens on Port 2379)

With that, your cluster also has a few requirements and these are:

Cluster string: This is a string of Comma-separated list of all cluster members in the format of name=peerURL. eg node1=10.10.10.0:2380,node2=10.10.10.0:2380
Cluster Token: This is a unique string to identify the cluster. It prevents instances from joining the wrong cluster.

Lets show an example of how to run a simple cluster. We shall run a cluster with 3 nodes having the following configuration:

Name	IP	Peer URL	Client URL
etcd-1	10.0.0.1	https://10.0.0.1:2380	https://10.0.0.1:2379
etcd-2	10.0.0.2	https://10.0.0.2:2380	https://10.0.0.2:2379
etcd-3	10.0.0.3	https://10.0.0.3:2380	https://10.0.0.3:2379

With the above instance settings, we can extract a cluster string and also generate a custom cluster token.

# Cluster String
etcd-1=https://10.0.0.1:2380,etcd-2=https://10.0.0.2:2380,etcd-3=https://10.0.0.3:2380


# Cluster token
my-cluster-token

Once last thing is that when you are initiating a cluster, there is another configuration you pass to etcd called the cluster initial state, this can either be new or existing.

Now since we have all the major configurations, we can then initiate our cluster by running each instance.

$ etcd \
 --name etcd-1 \
 --initial-advertise-peer-urls https://10.0.0.1:2380 \
 --listen-peer-urls https://10.0.0.1:2380 \
 --listen-client-urls https://10.0.0.1:2379,http://127.0.0.1:2379 \
 --advertise-client-urls https://10.0.0.1:2379 \
 --initial-cluster-token etcd-cluster-1 \
 --initial-cluster etcd-1=https://10.0.0.1:2380,etcd-2=https://10.0.0.2:2380,etcd-3=https://10.0.0.3:2380 \
 --initial-cluster-state new \
 --data-dir /var/lib/etcd \

You can run the same command another 2 times but adjusting the flags to match the settings of the other instances. Once that's done. You now have a running cluster.

Working with etcd

Now that we have installed etcd. We can move on to working storing and querying etcd

Interacting with `etcd`

When you install etcd, it comes with another binary called etcdctl. This is a CLI tool that helps you interact with the etcd API Server. However, there are other ways to interact with etcd and all include:

etcdctl: The official command line tool for interacting with etcd
HTTP API: etcd provides HTTP endpoints that you can make requests to.
gRPC Endpoints: Alternatively, there is also a gRPC API that you can interact with and its API reference can be found here
Client Libraries: Multiple SDKs, both official and unofficial exist and you can use these to interact with the API server.

Here is a list of client libraries for different languages that you can use in your project.

Language	Library	Maintainer	API Support	gRPC	Link
Go	`clientv3`	etcd-io (official)	v3	Yes	github.com/etcd-io/etcd/client/v3
Python	`python-etcd3`	Community	v3	Yes	github.com/kragniz/python-etcd3
Java	`jetcd`	Community	v3	Yes	github.com/etcd-io/jetcd
JavaScript / Node.js	`etcd3`	Community	v3	Yes	github.com/mixer/etcd3
C++	`etcd-cpp-apiv3`	Community	v3	Yes	github.com/etcd-cpp-apiv3/etcd-cpp-apiv3
Rust	`rust-etcd`	Community	v3	Yes	github.com/jimmycuadra/rust-etcd
C# / .NET	`dotnet-etcd`	Community	v3	Yes	github.com/shubhamranjan/dotnet-etcd

Querying etcd

Fetching and storing data in etcd is really simple. Partly because etcd is just a key value store. Lets go through the ways to query etcd

Storing data to etcd

Since etcd is a key value store, data is stored in keys and values, kind of similarity to dictionaries in python or objects in Javascript.

To store data, you just have to provide a key and the value of data you want to store.

etcdctl --endpoints=$ENDPOINTS put mykey "Hello World!"

That will store "Hello world!" into the mykey key. It will create a new key if the provided one does not exist in the database otherwise it will update the existing one.

Reading data

etcdctl --endpoints=$ENDPOINTS get mykey
>> Hello World!

Deleting data

etcdctl --endpoints=$ENDPOINTS del mykey

As I mentioned above etcd has language bindings(SDKs) for different programming languages. They provide utility functions to carry out these actions and manipulate your data. You can find out more operations that can be performed on data in the official documentation

Lastly, if you are building a golang application, you can embed theetcd server into your application.

How data is stored in etcd

When working with etcd, it is better to know how it works and how data is stored internally. This will help you in deciding how to store your data in etcd.

etcd uses a flat key-value data model, meaning there are no native hierarchical or nested data structures like JSON trees or SQL tables. Developers tend to use keys that represent some kind of hierarchy e.g. users/staff/jim. It might look like the data is stored in a hierarchy but it's not. A sample of how it might be stored internally looks like this.

{
 "/services/db/user": "admin",
 "/services/db/pass": "secret"
}

These keys look hierarchical, but etcd stores them as flat keys.

When data is stored, it is encoded in what is called protocol buffers on the disk. This is usually in the data directory which can be specified via the --data-dir flag when starting an etcd instance.

Internally, etcd uses a storage backend called BoltDB which is a fast embedded key-value store also written in Go, when data is stored in Bolt, it can be indexed using B-tree, which is a self-balancing tree data structure that maintains sorted data and allows fast lookups, insertions, and deletions. Each time data is mutated, it triggers an increment in an internal global revision. This means with etcd, you can read data at different revisions.

Operating etcd

Once you have an etcd cluster up and running, you will need to manage it to ensure maximum uptime, backups in case of node failures, debugging etc. As an operator you need to be familiar with the different tools and workflows required to manage etcd. Let's go through the common ones.

Cluster health

When running any system, you need to be able to observe the system health and status. etcdctl provides commands that can help you view cluster health. Simple run:

etcdctl --endpoints=$ENDPOINTS endpoint health


# You can add --write-out=table to have a table like output

Example

etcdctl --endpoints=$ENDPOINTS endpoint health


10.240.0.17:2379 is healthy: successfully committed proposal: took = 4.384885ms
10.240.0.19:2379 is healthy: successfully committed proposal: took = 3.938284ms
10.240.0.18:2379 is healthy: successfully committed proposal: took = 3.112832ms

etcdctl --write-out=table --endpoints=$ENDPOINTS endpoint status


+------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|    ENDPOINT      |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 10.240.0.17:2379 | 4917a7ab173fabe7 |  3.5.0  |   45 kB |      true |      false |         4 |      16726 |              16726 |        |
| 10.240.0.18:2379 | 59796ba9cd1bcd72 |  3.5.0  |   45 kB |     false |      false |         4 |      16726 |              16726 |        |
| 10.240.0.19:2379 | 94df724b66343e6c |  3.5.0  |   45 kB |     false |      false |         4 |      16726 |              16726 |        |
+------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------|

Snapshots

Snapshots are helpful in case of cluster failure, they enable you to be able to restore your data in case of an issue. Each snapshot can only work on one etcd instance at a time, not the whole cluster. So when creating a snapshot, ensure that you set the --endpoints flag to the endpoint of only one instance.

etcdctl --endpoints=10.240.0.17:2379 snapshot save backup.db


>> Snapshot saved at backup.db

Monitoring etcd

etcd is designed to be easily monitored and it follows industry standards.

Metrics

etcd exposes Prometheus compatible metrics that can be pulled in realtime. However these metrics are not persisted across node restarts, so once a node restarts, the metrics counters also restart.

etcd exposes its metrics to the /metrics endpoint and you can spin up a Prometheus server and configure it to pull metrics from that endpoint.

Logging

etcd uses standard logging formats, it internally uses zap for logging. THis means all logs contain metadata about the log levels to understand what each line is about. It provides 5 logging levels. Error(for errors that occur), Warning(for temporary but important conditions that may cause errors), Notice(for important information), Info(for normal logs in the program flow) and Debug.

An etcd log can look something like this.

{"level":"warn","ts":"2025-07-24T16:58:58.804311+0300","caller":"embed/config.go:1209","msg":"Running http and grpc server on single port. This is not recommended for production."}

Conclusion

Hopefully this article has given you a good understanding of etcd and you can setup, interact and operate a simple etcd cluster. However, you learn by doing so i would suggest you set up your own cluster and play around with it. Etcd has a wide range of use case scenarios and to even gain the full power of this system, I would recommend trying to use a programming client, preferably the official Go client to interact and use etcd. That being said, that's all I had for today. Thanks for reading.

Designing The Conveyor CI Pipeline Engine

Beingana Jim Junior — Tue, 22 Jul 2025 10:02:54 +0000

In CI/CD Systems, there is a concept of pipelines. Pipelines define the steps of how code changes throughout the entire CI/CD process. Multiple systems like Github Actions, GitLab CI/CD, Jenkins etc all have this functionality of pipelines natively embedded within them, something that Conveyor CI currently lacks. You could engineer a walkaround to implement your own pipeline-like functionality in Conveyor but natively, this functionality does not exist and this is a serious downside existing within Conveyor.

This got me thinking, how could I implement it. As you might guess from the name, Conveyor CI was inspired by how a conveyor system in an industry works. Take an example of a car manufacturing factory, the skeleton of the car moves along the conveyor system and at each step there is a dedicated robot in charge of attaching a certain component to the car to carry out a certain function. By the time the skeleton reached the final stage, its a complete car. From the start I wanted to adopt this kind of concept as the core paradigm of Conveyor CI whereby a resource(e.g. source code) moves on a belt and at the end of the belt, all necessary actions are carried out.

With this in mind, I split the Conveyor CI into components in order to easily understand it adopting from components of a conveyor system ie. Package, conveyor belt, and Peripheral equipment. In that I came up with the following:

Resource: The resource is an internal object in Conveyor that represents what is being acted upon throughout the CI/CD process. Think of it as the Package on a conveyor system. It can represent anything from source code, and application, a program etc.
Drivers: Drivers are software components that carry out certain actions depending on the state defined in the Resource. Think of them as the peripheral devices that act on a package as it moves along a conveyor system. In this case the Packages are the Resources.

Notice I have not mentioned a component that corresponds to the conveyor belt. That is because currently there was none, and to create one was where pipelines come in.

So currently in Conveyor CI we have Resources and Drivers. They are generally mature enough as individual components and although you could create an entire CI/CD tool with these only, there are still some issues. Mainly is that there is no order of execution of drivers upon a Resource. This means that once a Resource event occurs, all drivers execute their corresponding actions whenever they receive the event and they do this in no orderly fashion. This means it is not possible to predict what action will occur at what point in the CI/CD process and you also can’t create dependency actions that depend on others being pre-executed. An example is if you have a workflow whereby you compile your source code then upload the output program to a distribution server, You might have two drivers, one for compiling and another for uploading. In the current implementation, these drivers will execute at once, yet the uploading driver should depend on the constraint that the compiling driver is done executing.

To fix this we have to come up with a way to define a workflow that drivers must follow when carrying out their executions, something that defines the steps and order that these drivers will follow throughout execution. Something to act as the conveyor system. This is where Pipelines Come in. They will define the order of execution followed by drivers. The pipeline can be an object containing the configuration defining the order followed by drivers.

Pipelines also introduce more possibilities like shared context among driver executions, meaning, a pipeline can define some metadata that is shared and used across all the drivers that are executing upon a resource in that pipeline.

Implementing Pipelines

Now that I had come up with a high level design of what pipelines are expected to work, I had to move on to designing an implementation that would easily fit in into the existing Conveyor CI implementation. Inorder to create a implementation that is good, i set a few constraints that i had to follow:

Pipelines should not introduce breaking changes that might interfere with the Developer experience and development paradigm for developers developing drivers. It should be more of an incremental change, building upon the already existing development paradigm without breaking already existing codebases.
Minimize the risk of over engineering the system while trying to follow constraint one. This is because usually one trade off of maintaining a good developer experience is that you might end up over engineering the system.

With these design constraints in mind, I embarked on designing the Pipeline engine. First I started with the usage workflow from the user viewpoint i.e. the process that will be followed by a developer when creating a pipeline and using it.

Use case Workflows

Considering that developer experience is one of the most important concepts for the success of a tool, I had to come up with a really easy to understand workflow and also not break the already existing usage workflows in Conveyor CI. I came up with this.

First a pipeline is designed by the developer choosing their appropriate drivers and resources and arranging them in a desired order.
The pipeline is then sent to the Conveyor CI API Server to be registered and saved.
Once its saved then a Resource that is using that pipeline is created and sent to the API server
Then the Pipeline Engine appropriately routes the resource to the drivers.
Driver do there work

How the Engine Works

With this in mind, I came up with a system design that can accomplish these functions but also act as an incrementation on the existing Conveyor base.

First a new state object called a Pipeline was introduced into the system. This object would store and represent information about a pipeline throughout the stages of execution. It would store the order of execution followed by drivers and additional context information/metadata that is required by the driver in said pipeline.

I also had to introduce a Driver Result event object. This object, acting as an event, was to be used by drivers to communicate the result of the Reconcile function(The function that is run when a driver receives a resource to act upon). This would be used to inform the Pipeline Engine if a desired driver operation was successful or not.

In order to maintain realtime seamless execution, a new execution process running concurrently with the API Server had to be introduced to Conveyor CI's core runtime program. This would be in charge of listening for new resources attached to pipelines, routing those resources to drivers in a pre-defined order as defined in the Pipeline objects, and watching for pipeline realworld state changes and reconciling it to the state stored in the Database. This process is what I named the Pipeline Engine

Execution Workflow

Having introduced the required components, I came up with this workflow.

The user sends a POST request to the Conveyor API server to register their Pipeline, this is then saved to the ETCD Database.
Then the user creates a pipeline Resource(a resource that depends on that pipeline), also via the API Server.
When the API server receives and saves this Resource to the DB, it sends an Event to the Pipeline Engine.
The Pipeline Engine receives the Event and using the event metadata it collects the Pipeline information from the DB and depending on the driver execution order defined by the pipeline it begins to send events to the drivers.
Once the Driver has finished to carry out its functions, it returns a Driver Result and the Driver Manager sends that result as an event back to the Pipeline Engine.
When the Result event is received, the Pipeline engine can then move on to sending the event to the next driver. If no result event has been received yet from the Driver, the Engine wont move on to the next Driver.
Lastly, if the result event on one driver indicates that the Driver execution failed or an error occurred. The pipeline engine won't send events to the remaining drivers and will rather register the pipeline as stopped or done with a status of failed. Else it will finish execution of all drivers and still register the pipeline as done/complete.

A Simple Diagram representing this process

Technical Details

I have managed to describe the high level working of how the Pipeline engine would work. But beyond that, integrating such functionality into the Conveyor Stack requires clarification on how to implement the different components technically.

Starting off with the Pipeline engine itself, we said it's a process running concurrently with the API Server in the same program. Considering that the Conveyor CI core software program is written in the Go programming language and Go as a language has inbuilt concurrency via the Go routines, this means that the Pipeline Engine would better be a go routine running alongside the API server on the same process. One upside of this mechanism of embedding the Pipeline engine into the same process with the API Server is that we will require little updates to the metrics and monitoring codebase in order to add metrics for monitoring the Pipeline Engine.

Inorder to achieve the realtime nature of execution among these separate execution runtimes, we utilize events via NATS Jetstream, the message broker that is already being used by Conveyor. This means that the Pipeline Engine performs both the roles of an Event publisher and subscriber. When new Pipeline resources are created, an event is sent to the Pipeline engine via JetStream. In this scenario, the Pipeline Engine is running as a Jetstream Consumer, listening for events via the pipeline stream. Upon receiving an event, it will collect the pipeline data from ETCD and collect all the driver names. Using the driver names it will now become an event publisher and publish to the messages stream(the default stream in which drivers listen for events) and use the subjects with the following semantics {DRIVER_NAME}.resources.{RESOURCE_NAME} with the placeholders {DRIVER_NAME} and {RESOURCE_NAME} referring to the name of the driver to publish to and the name of the resource being published respectively. All we have to do now is update the driver managers of the drivers, to include the new subject in the filtered subjects field of there Jetstream consumers. This will require updating all the SDKs though. With this, we have a complete working Pipeline engine.

Finally to keep track of Driver executions that belong to one pipeline, we utilize an already existing concept in Conveyor CI which is the Run ID. This is a UUID string that identifies individual driver runs/executions upon a resource. In a pipeline, we maintain the same run ID throughout executions in a single pipeline execution.

Conclusion

The whole point of Pipelines was to introduce ordered execution of drivers on a resource but this feature can act as a fundamental building block for more robust and important functionality to be integrated into Conveyor CI. With this I have to acknowledge that Conveyor CI is still a really immature project, it really still has a long way to go and still lacks important features e.g. resource versioning etc. But one step at a time. Just like Rome wasn't built in a day, You can build a Cloud native software framework in a short period of time.

A Guide to contributing to the Conveyor CI Driver Runtime

Beingana Jim Junior — Fri, 20 Jun 2025 20:38:10 +0000

A detailed guide to contributing to the Conveyor CI Driver runtime.

Introduction

A brief introduction on the Conveyor CI Driver runtime.

The Conveyor CI Driver runtime is a collection of SDKs(Software Development Kits) used by developers to build Conveyor CI Drivers. These SDKs are libraries that contain Utility functions that expose different Conveyor CI functionality. In this Guide we shall explore how one can contribute to the Conveyor CI Driver Runtime. This Guide is written to be language agnostic, meaning it can guide you to build or contribute to a driver runtime of your language of choice.

How the Driver Runtime Works

Lets explore what the Runtime is

What is the Driver Runtime

You can think of the Driver runtime as the wrapper of Driver applications and is the environment in which Drivers execute. You can define it as the software layer that provides necessary services, infrastructure, and behavior for executing the driver. Think of how language runtime work like the JVM, Go Runtime, V8, CPython etc. If you where to write a program in those languages, run in an environment that contains your code and the language runtime.

So even when building a Conveyor CI Driver, the final program is a combination of the Driver runtime and your custom Driver code.

What Features does it Provide

As mentioned earlier, the driver runtime provides necessary services and utilities for building CI Drivers. Some of these features are not just exclusive to the runtime but rather and extension of the base features provided by the Conveyor CI Ecosystem. These features include:

A mealtime Event System: The driver runtime is designed to ensure drivers receive and publish events within the Conveyor System to ensure quick and instant execution of tasks.
Log Management: The Driver runtime provided a Driver Logger, that provides drivers with functionality to export and save necessary logs in real-time. These can then be fetched or streamed in real-time or for future use.
Horizontal Scaling: The Driver runtime provides out of the box Horizontal Scaling to Drivers, meaning once you build a driver. It comes with the ability to horizontally scale into a distributed system efficiently without you writing any custom code. This feature can be really important in Cloud Native environments.
Conveyor CI API Server Interaction: No driver runtime package is complete without utility functions to interact with the Conveyor API Server to for example create and manipulate Resources.
Observability out of the Box: The Driver runtime is designed to provide a comprehensive observability platform that follows all the three pillars that include Metrics, Tracing and Logging.

And many more...

Driver runtime Components

The Driver runtime contains mainly two components. The Driver Manager and the Client Library.

Driver Manager

The Driver manager, the component that encapsulates the Driver and provides utilities that are used by the Driver. By design, it is usually a Class or a Class-like data type that has a run() method that is called to start the program. Its is responsible for the following functionality.

Listening for events from Conveyor CI sent via NATS Jetstream: This is accomplished using JetStream Streams and Consumers. The Driver manager contains a Consumer that listens for events from the messages stream and filters out subjects depending on the resources that the Driver defines to listen to.
Provides the Driver Logger to the Driver. The driver logger is a component that collects logs from the driver and send them to Grafana Loki for Storage. After storing them it also streams them via a NATS connection on the subject with the following semantics driver:{DRIVER_NAME}:logs:{RUN_ID} with the place holders DRIVER_NAME and RUN_ID referring to the name of the driver and the current run id respectively.
Runs the Reconcile function of the Drivers upon each event from JetStream and passes in the appropriate parameters.

Driver Logger

The Driver logger component is in charge of collecting logs from the Driver, sending them to Loki Grafana for storage and also streaming them via the NATs connection. It accomplished this by following the rules stated above in the Driver manager section

Driver

The Driver is the main component that developers interact with. It contains the Reconcile function that contains the custom code that includes the drivers intended functionality. When initializing it, the constructor must require the developer to specify the Resources that the driver wants to control, the driver name and also the Reconcile function. The Reconcile function, is a function that takes in these parameters. payload, event, driverName, logger.

Client Library

The client Library is a component of the driver runtime that contains functions that interact with the Conveyor CI API Server. these are normal HTTP Requests to the API Server. It also has the ability to fetch Conveyor CI API Server metadata like the API Host and Port.

Note: The Class Diagrams above represent the general expected Structure of those components however due to differences is language syntax and semantics in programming languages. The structure can be tweaked to provide the Developers a good user experience.

Component Diagram

Contributing Guide

Now that you have a solid understanding of the high level structure of the Driver runtime. Lets move on to the practical guide of the Workflow of Contributing to or building a driver runtime SDK package. Shall first go through the Development Environment setup then also go through the general contribution workflow followed when contributing to the source code of official Conveyor CI Driver runtime SDKs.

Environment Setup

To setup your environment to run conveyor, all you need is to have Docker installed on you computer. You can head over to the Docker Installation Page to install it on your system.

Once you have it, you will need to download two configuration files that are used to run Conveyor API Server and System dependencies. Head over to the Conveyor CI Releases Page and on the assests of the latest release, download compose.yml and loki.yml and store them in the same directory. Or you can simply run these commands.

curl -s https://api.github.com/repos/open-ug/conveyor/releases/latest | grep browser_download_url | grep compose.yml | cut -d '"' -f 4 | xargs curl -L -o compose.yml

curl -s https://api.github.com/repos/open-ug/conveyor/releases/latest | grep browser_download_url | grep loki.yml | cut -d '"' -f 4 | xargs curl -L -o loki.yml

Once you have both those files in the same directory, you can then start the Conveyor CI Containers by running.

docker compose up

# OR

docker compose up -d

Take note of the ports and host that these containers run no, or you can just open the compose.yml file and see what ports are being exposed so when you are developing the packages, you are aware what ports and host to connect to. Also note that in-case its a requirement to set different Ports or you want to configure your own ports. always follow the convention of setting environment variable CONVEYOR_SERVER_HOST and CONVEYOR_SERVER_PORT

Once you have everything running your are good to go onto development.

Workflow

We welcome contributions of all kinds, Whether you're fixing a typo, reporting a bug, improving documentation, or adding a new feature, your help is appreciated. However when contributing to the Driver runtime, you need to follow a specific contribution workflow.

There are multiple kinds of contributions we follow and they include

Reporting Bugs

If you find a bug, you can help fix it by submiting and issue to the appropriate repository. within your issue Include:

A clear and descriptive title
Steps to reproduce the issue
What you expected to happen
What actually happened

If the repository has an issue tempate. you should follow it.

Before submitting, check if the issue already exists in the repository issue list.

Suggesting Enhancements

We also welcome feature suggestions and ideas for improvement. When submitting a feature request:

Explain why the feature is useful
Provide example scenarios where it would help
Suggest a possible implementation, if you have one

Try to keep requests focused and concise.

Submitting Code Changes

Once you have identified and issue or an enhancement you would like to work on. you can folloe this workflow to submit your code changes.

Fork the repository
Clone your fork:

   git clone https://github.com/your-username/project-name.git
   cd project-name

Create a new branch:

   git checkout -b feature/your-feature-name 

   # OR 

   git checkout -b fix/your-fixture-name

Make your changes
Write or update tests, if applicable

Commit your changes with clear messages:

git commit -m "feat: add new feature X"
# OR
git commit -m "fix: solved issue Y"

Push your changes:

git push origin feature/your-feature-name

Open a Pull Request on the GitHub repository and describe what you’ve done

Pull requests should be:

Focused on a single change
Thoroughly tested
Aligned with project’s code style and guidelines

Improving Tests or Documentation

Improving test coverage and documentation is highly valuable. You can:

Add test cases for untested components
Update outdated documentation
Fix typos or formatting issues

No contribution is too small!

Resources

A few resources that you might find useful when trying to understand the Conveyor System

NATS AND JETSTREAM RESOURCES

CONVEYOR GO PACKAGE AND DRIVER RUNTIME

Repository

OTHER RESOURCES

Designing the Conveyor CI Scheduling Mechanism

Beingana Jim Junior — Mon, 16 Jun 2025 17:08:55 +0000

A brief introduction on Conveyor CI. Its is a Software framework fdesigned to simplify the development of scalable and reliable CI/CD platforms by eliminating common pain points that a team ight to run into inorder to build a Scalable, reliable CI/CD Platform. It provides a simplified streamlined, developer-friendly development paradigm for building robust custon CI/CD Tools.

Recently a team that uses Conveyor CI as a dependency had an inquiry on how a CI Driver in conveyor can handle multiple instances of running CI processes without overloading the host node. This is a really important consideration I had not thought of, and for a system that runs multiple tasks in concurrency, being conscious of system resources is a really important consideration.

This got me thinking, how best can one approach such an issue. I thought of multiple solutions to this, I had considered introducing a consensus algorithm like raft, however this wasn't really an effective solution. Looking into how the Kubernetes Kube Scheduler efficiently allocates Pods to run on different nodes taking into considerations or the resource constraints of both the Pod to be allocated and the nodes its to be allocated to gave me an inspiration of having to design a custom scheduling algorithm that had to to take into consideration of the multiple, use case scenarios of CI/CD tools and what constraints they operate under.

I had to first come up with the high level requirements and problem statements for the CI Drivers in order to tackle this issue.

The Problem: In large production grade CI/CD Pipelines, multiple requests can initiated at the same time and due to the underlying concurrent and instant execution of CI tasks by Conveyor CI Drivers, this introduces an issue of possible node crashed or driver process termination by the Operating system due to the host node running out of system resources. Also currently, drivers don't support horizontal scalability, something that is really important for CI/CD tools that are expected to run multiple workloads at a certain point in time considering vertical scalability tends to be finite and also costs tend to increase exponentially the more you scale vertically.

From that problem statement I managed to identify two high level requirements.

Drivers should support horizontal scaling
The Driver runtime should take into consideration available node resources before executed a task

After a long night of insightful research, I came up with a bunch of solutions for this, these would assist me in designing the scheduling algorithm. I would call the functional or technical requirements if that's the right term to use.

The algorithm was to take inspiration and be a combination of multiple OS scheduling algorithms, with the help of a consensus algorithm to handle and support the distributed properties of the Driver instances.

To solve the issue of the possibility of overloading the server with tasks that require system resources than available, First i had to remove the instant execution of tasks and introduce a task queue, this would enable me to be able to incorporate a FIFO-like architecture as the base algorithm for scheduling tasks in a normal flow. Taking into consideration that in certain organisations, CI/CD tools are designed with the concept of having some tasks prioritized due to multiple factors, say if they would like to prioritize tasks from paying users. I also decided to incorporate a Priority Scheduling Algorithm as the second level algorithm. Lastly, I had to introduce a constraint based scheduling algorithm as the last determining algorithm, these constraints can be the available system resources etc. To support such a complex algorithm, I had to introduce heuristics that are computed by the Driver from a resource specification or defined directly in the resource specification. These heuristics can be task priority, available node resources etc.

On the issue of horizontal scalability, I decided to incorprate NATS Jetsream Consumers. This would enable distributed processing of Driver tasks laveraging the already existing powerfull features that NATS Jetstream offers out of the box.

How the Mechanism works

Alright, let's get into how it works. At the core, the tasks are organized in a priority queue, where tasks with a high priority are executed first and in instances where the priority number is equal, they are executed based on a FIFO order based on a timestamp.

Preventing resource contention

Within the tasks Resource specification, I introduced an optional field called resourceUsage that defines the expected resource usage for the task during execution. This acts as heuristic that helps the driver runtime decide whether a certain driver instance is allowed to run the task. It compares the available host node system resources and evaluates whether running that task will not cause resource contention or Out of Memory Errors. This constraint based scheduling considers any node as long as adding the task won't exit 100 percent usage of the resources, that's to say, even if a node is using up to 90% of its resources and adding the task will push it to 99% it will still schedule it on that node. it also tends to prioritize driver instances running on nodes with lower resource usage than the others, say if node A is using 60% and node B is using 55%, it's more likely to consider node B than A. But this will not always be the case, sometimes other factors might cause us to consider node A. Such a mechanism will ensure that the possibility of node failures due to resource contention is really low.

Here is some Pseudo code example to demonstrate the algorithm structure

Initialize priorityQueue ← empty priority queue (higher priority first, FIFO if equal)
Initialize nodeList ← list of available nodes with current resource usage

Function scheduleTasks():
    while priorityQueue is not empty:
        task ← priorityQueue.dequeue()
        eligibleNodes ← []

        for each node in nodeList:
            if node.canAccommodate(task.resourceUsage):
                eligibleNodes.append(node)

        if eligibleNodes is empty:
            priorityQueue.requeue(task) # Retry later
            continue

        Sort eligibleNodes by resourceUsage (ascending)

        selectedNode ← selectBestNode(eligibleNodes, task)

        # Assign task to selectedNode
        Update selectedNode.resourceUsage += task.resourceUsage

Function node.canAccommodate(resourceUsage):
    return (node.resourceUsage + resourceUsage) ≤ 100%

Function selectBestNode(nodes, task):
    # Prefer lower resource usage, but allow for flexibility
    # Example heuristic: select node with minimal usage
    return nodes[0] # or apply more complex scoring if needed

Function addTaskToQueue(task):
    priorityQueue.insert(task) # uses (priority, timestamp) as sort key

Horizontal Scaling

When it comes to horizontal scaling, I decided to re-engineer the Drivers to run as a distributed system in cases where there are multiple driver instances running. This distributed system is powered by NATS JetStream Consumers

At the core each Conveyor CI Driver operates as a JetStream Consumer. They listen to a stream which filters out required messages and pushes them to the driver. The driver then carries out the required tasks depending on the payload recieved from the stream.

The Drivers are configured to use Explicit acknowledgement with the value set to 1 so that each task it handles one by one and this would prevent a possibility of one driver handling multiple tasks at once possibly causing resource contention. Acknowledgement occurs once the message is recieved by the driver. We also set the Delivery policy to All, so that no task is skipped and all tasks are handled. Another feature that Jet

By utilising JetStream, it enables driver to be able to infinitly scale horizontally and become more failt torolent by utilizing various inbuilt NATS features.

Conclusion

Designing a robust scheduling algorithm for Conveyor CI Drivers was a necessary and will definetely improve the functionality and reliability of the Drivers when the scheduling mechanism is added to the Conveyor CI codebase. By combining FIFO, priority-based, and constraint-aware scheduling with a distributed model powered by JetStream, I’ve managed to create a system that not only prevents resource overloads but also opens the door for true horizontal scaling and more features like fault tolorence. This isn’t just a theoretical improvement—it directly addresses real-world pain points for teams running concurrent workloads on shared infrastructure. There’s still room to refine and optimize, but this foundation sets the stage for building CI/CD systems that are both resilient and adaptive under pressure. This might be the greatest improvement to Conveyor CI yet.

How to deploy an anonymous website on the Dark Web

Beingana Jim Junior — Wed, 19 Mar 2025 19:02:16 +0000

Have you ever wondered what the Dark Web is? How to access it? Or even better, how to deploy your site to the dark web? Well I got you today. In this article we shall look at how you can deploy your site on the Tor network, which is a Dark web that allows clients and Servers to navigate and exist on the Internet Anonymously.

The Internet is not all you see, it's divided into 2 parts, The Clearnet and The Darknet. They are differentiated by Traceability. You see on the Clearnet, when a message is sent over the network, it's easy to trace the client that sent it and the destination server that it was sent to. That's not the case on the Darknet, on the Darknet the location/IP address of the server and the client are not traceable. The client sends a message to the server but doesn't know where it's located.

Take an example of youtube.com, If we do an NSLookup, we can identify the IP Address, Location and ISP of the server, in this case as of now its, 142.251.12.190 with location in Singapore and Google LLS ISP.

This traceability is due to the underlying design of the TCP/IP Protocol where communication, whether using TCP or UDP for communication, the IP address of the server and client is traceable and known.

Even in cases where we try to hide our identity over the internet, there is still some form of traceability. In large organisations, they usually try to make the client address private and all traffic is routed through a NAT. This keeps the address of the client hidden but the IP Address of the NAT is public to the Internet. Another way we attempt to navigate the internet anonymously is using VPN Systems, these work by routing the traffic from the client through a relay system or server and then this routes the traffic to the destination server. However much the IP Address of the client does not make it to the mainstream Internet, that of the VPN System is public to the Internet. Generally, there is always some form of traceability on the Clearnet

On the Darknet this traceability is not there, Darknets are designed with confidentiality, privacy, and freedom in mind, so maximum anonymity is ensured. Notice i just said "Darknets", this is because there are more than one form of Darknet. Actually the term "Darknet" refers to any overlay network that requires special software configurations and authorisation to access and is not indexed by search engines. Some forms of darknet technologies include I2P (Invisible Internet Project), Freenet, ZeroNet etc. But the most popular so far is Tor (The Onion Router) and its the one we shall be discussing and using in this article.

The Onion Router (Tor)

The Onion Router, often known as Tor, is one of the forms of darknet technologies that is under operation and so far the most popular. It runs on a core technology known as Onion Routing that was developed in the mid 1990's by the United States Naval Research Laboratory to protect American intelligence communications online. In 2004, the Naval Research Laboratory released the code for Tor under a free license and now runs on free and open-source software and more than seven thousand volunteer-operated relays worldwide.

It is built with the principles of confidentiality, privacy, and freedom in mind. It enables anonymous communication over the internet between clients and servers. To accomplish this, strong encryption algorithms/technologies like SSL/TLS and relay servers are utilised to prevent message Man-in-the-Middle (MITM) attacks and Traceability respectively.

How it works

Let's take a look at how Onion Routing works. In Onion Routing, a message is encapsulated with multiple layers of encryption, analogous to the layers of an onion, this message is then passed through a series of network nodes/relay servers known as "onion routers" that peel of a layer of encryption one by one at each node until the message arrives at the destination server. This mechanism provides confidentiality. THe diagram below shows how these encryption layers are peeled off at different nodes in the network

When a client sends a network request over the Tor network through the Tor Browser (The web browser used to browse the Tor Network). The browser first sends the request to a gateway node known as the Entry Guard. This is the Entry Point into the Tor Network, however the Entry guard is not aware that it is the Entry point of the network and also whether the message came from another relay node or from a client. The Entry guard then connects to another relay called Middle Relay which also connects to another Middle Relay. The last relay in the chain is the Exist Relay which connects to the website. Each relay knows only the previous system and the next one, but never the whole chain. In the point of view of the website, it's being accessed through Tor and is aware of the address of the Exit Relay but cannot know anything else beyond that. This chaining of relay servers is what enables Tor to eliminate traceability.

One last thing we need to know when accessing a hidden service is the Onion Address. Unlike normal websites which we access using their URLs, hidden services are accessed using a special type of addresses called onion addresses. Each Onion Address contains an opaque non-mnemonic string of 16 characters followed by a special TLD (top level domain) address of .onion. The .onion addresses are not part of the typical DNS on the Internet hence are not accessible over traditional internet clients. THis is one of the reasons why Tor has its own special browser. Here is an example of an Onion Address: http://zsuwmakpr3gfryt6.onion

Deploying on the Tor Network

Now that we have a general understanding of how the Tor network works, let's move on to deploying a simple website to the Tor Network. We shall do this by following a series of steps. Before continuing ensure you have a Linux server or Virtual Machine running.

Installing Tor Services

To use the Tor network, you are required to have two main components, The Tor Browser and the Tor Server. The Tor browser can be installed on almost all platforms, head over to the Tor Download page to download it for your platform. On the other hand, as far as I know The Tor server is a linux program so you are required to have a linux Instance running. If you're using a debian system run the following commands to install it.

apt-get update
apt-get install tor
apt-get install torbrowser-launcher

Setting up your Web Application/Server

Depending on the tech stack of your application or website, you will be required to start your website on the local host address (127.0.0.1) listening at a specific port, for example port 3000. Ensure that the port you're listening to is not exposed outside the server as this will make your site accessible over the Clearnet hindering the anonymity we are aiming for.

Add the Website to Tor

Now that we have our web server listening on localhost, we can configure our Tor program to designate this server as a Tor hidden service. We do this by editing Tor configuration file /etc/tor/torrc as follows:

############### This section is just for location-hidden services ###

## Once you have configured a hidden service, you can look at the
## contents of the file ".../hidden_service/hostname" for the address
## to tell people.
##
## HiddenServicePort x y:z says to redirect requests on port x to the
## address y:z.

HiddenServiceDir /var/lib/tor/hidden_service/
HiddenServicePort 80 127.0.0.1:3000

When you open the torrc file, you will find the lines HiddenServiceDir and HiddenServicePort commented, so you will have to uncomment them and also modify the HiddenServicePort to make sure that it points to the port at which you application is listening.

Running the Tor Service

After setting up the required configuration, we can then run the Tor Service to perform all necessary actions and register our site to the Tor network. It will also create private and public keys for encryption and also create an onion address for your site. To run the Tor Service, simply run the command tor

$ tor

Nov 13 17:18:09.832 [notice] Tor 0.3.4.8 (git-5da0e95e4871a0a1) running on Linux with Libevent 2.1.8-stable, OpenSSL 1.1.0h, Zlib 1.2.8, Liblzma 5.2.2, and Libzstd 1.3.5.
Nov 13 17:18:09.851 [notice] Tor can’t help you if you use it wrong! Learn how to be safe at https://www.torproject.org/download/download#warning
Nov 13 17:18:09.947 [notice] Read configuration file “/etc/tor/torrc”.
…
…
…
Nov 13 17:18:36.000 [notice] Bootstrapped 90%: Establishing a Tor circuit
Nov 13 17:18:37.000 [notice] Tor has successfully opened a circuit. Looks like client functionality is working.
Nov 13 17:18:37.000 [notice] Bootstrapped 100%: Done

You can also use a service like nohup inorder to run it as a background process

nohup tor &

The Tor program will create two files under the directory /var/lib/tor/hidden_services/ which include the Private Key and a hostname file containing the onion address of your site. Ensure to keep the Private Key secure and confidential.

You can now get your onion address by reading the contents of the hostname file

cd /var/lib/tor/hidden_service/

cat hostname

zjsytjmah6hiyw1h.onion

Lastly, open the Tor Browser and head to the onion address provided and you will be able to access your anonymous site.

The Dark Web is often known for the illicit Activities that are carried out over it. This has portrayed the Dark Web as a dangerous place on the Internet. However, the Dark Web is used for more than that. Law enforcement agencies and cybersecurity professionals use Tor for undercover operations, intelligence gathering, and security research to combat cybercrime, terrorism, and fraud. Researchers, companies, and individuals dealing with sensitive topics (such as cybersecurity, medical research, or financial transactions) use Tor to protect trade secrets, proprietary data, and confidential communications from cyber threats and corporate espionage. Human rights activists and political dissidents use Tor to evade surveillance and protect themselves from oppressive governments that seek to suppress dissent. This fosters democracy and civil liberties worldwide.

An Introduction to Supervised Learning Models

Beingana Jim Junior — Sat, 08 Mar 2025 14:21:29 +0000

Machine learning is a subset of AI that learns to make decisions by fitting mathematical models to observed data. This area has seen explosive growth and is now (incorrectly) almost synonymous with the term AI. Machine learning models are generally categorized into two. Supervised and Unsupervised learning models. In this article we shall take a look at Supervised Learning models, how they work and different concepts associated with them.

What is Supervised Learning

Supervised learning is a field of machine learning where mathematical models are trained on labeled data from datasets using the input and expected output inorder for the model to learn the relationship between the input and output. That aim is for the models to be able to predict data from new unseen data.

Problems in Supervised learning

Problems in Supervised learning are usually classified into two kinds. Supervised learning models primarily address two types of problems. Regression and Classification models.

Regression problems are problems that require a model that predicts continuous values. Think of a problem which requires us to predict the price of a house, we are to create a model that takes inputs which are various characteristics like location, size etc, and from these inputs we predict a price. We can also have problems that predict more than one value and these are called multivariable regression problems. Take an example of a model that predicts the age and height of a tree.

Classification Problems on the other hand are problems that produce a categorical output, for example a model that predicts if it will rain or not. This is actually called a binary Classification problem since the output has only two states to exist. However, the model can predict an output that can be in one of more than 2 possible categories. These are called multiclass classification problems. Take an example of a model that predicts what animal is in a picture, and the animal can be anything from a dog, cat, cow, goat etc.

Supervised Learning Models

In supervised learning, we aim to build models that take an input $x$ and predict an output $y$ . The model is a mathematical equation $f [•]$ that takes inputs $x$ and predicts $y$ . We can then represent the output as $y = f (x)$ .

This process of computing a prediction $y$ from $x$ through the model is called inference

The model is a fixed equation that contains parameters $θ\theta$ , the choice of parameters $θ\theta$ determines the relationship between the input and prediction. Therefore the prediction $y$ varies based on the parameters. We can then rewrite the equation to support the parameters $θ\theta$ as.

f[x,\theta]

When we say we are training a model, we are feeding the models with input output pairs $(x, y)$ and aim to find the parameters $θ\theta$ that match inputs $x$ to outputs that are as close as possible to the expected output $y$ .

The mismatch between the output and expected output is called the loss $L$ . This loss is a scalar value that describes how poorly our model predicts the expected output. We can treat the loss as a function $L(θ)L(\theta)$ of the parameters $θ\theta$ . Generally we are training the model to find the parameters that minimize the loss as possible.

After training a model, we must now assess its performance; we run the model on separate test data to see how well it generalizes to examples that it didn’t observe during training. If the performance is adequate, then we are ready to deploy the model.

Model Inputs

Learning models are usually mathematical equations that take inputs in numerical form, but in the real world, these inputs appear in a variety of formats, so we have to find a way to encode them into numerical form.

Inputs and how they are interpreted can vary in different forms, for example if we are predicting the price of a house, the inputs are location, number of bedrooms etc and it doesn't matter the order in which they are input, the model is expected to predict the same price. Now take an example of a model that predicts if a sentence is realistic or not, the sentences, "my wife ate the chicken" and "the chicken ate my wife" are the same in terms of the words they consist of but the order is different hence the meaning and the expected model prediction is different.

Model Outputs

The output of the model also varies wildly depending on the type of the problem. Although we said the problems being solved are regression or classification problems, the output in real world applications is not usually a single value (as in classification or regression) but a structured object, such as a sequence, tree, or graph. For example a Speech recognition model that transforms audio waveforms to a sequence of words.

Conclusion

This is just an overview of Supervised learning models, in the next article, we shall look at a Practical example using Linear Regression. However, if you want to learn more about supervised learning models, I would recommend Reading Chapter 2 and 3 of the Understanding Deep Learning Book.

Thank you for taking the time to read this article. I hope it has been informative. About me, I am Beingana Jim Junior, a software engineering student at Makerere University. Check out my Github and Portifolio website

Feel free to share your thoughts, questions, or feedback

A Practical Introduction to the Event Driven Architecture

Beingana Jim Junior — Sat, 18 Jan 2025 14:03:41 +0000

After working extensively with microservices, I’ve come to appreciate the power and flexibility of the Event-Driven Architecture (EDA). EDA is a software design pattern that allows systems to detect, process, and react to real-time events in a decoupled manner. It is particularly suited for microservices, enabling seamless communication and interaction between independent components.

In this article, we’ll explore the principles of event-driven architecture, discuss its core concepts, and demonstrate how it works using a custom-built command-line tool. This tool, written in Go and powered by Redis, serves as a practical example of EDA in action.

To follow along with the demonstrations, you’ll need the following:

Download the CLI tool from GitHub (available for Windows, Linux, and macOS) Make sure to download the latest release.
Have a Redis server running locally. You can install Redis from the official website or use Docker for easier setup (recommended):

How the Event-Driven Architecture Works

At a high level point of view the EDA is quite straight forward. Events representing occurrences or changes in the system drive the flow are generated by various sources (Publishers), published to an event bus or message broker, and consumed by interested components (Consumers) asynchronously. This approach promotes flexibility, scalability, and resilience.

An event is any change in state of the system and can be anything ranging from customer requests, inventory updates, sensor readings, and so on.

The event-driven architecture enables different components of a system to run independently of each other by introducing a middleman known as an Event broker that routes events to different intended destination components. This means that applications and devices do not need to know where they are sending information or where the information they are consuming comes from. Rather they focus on carrying out there intended functionality.

Note: In this article we use Redis because of its simplicity and realatively easy learning curve but Redis wont demonstrate the full functionality of an Event Driven System, however there are other more complex software systems such as RabbitMQ or Apache Kafka that provide a wide range of functionality and Protocals such as AMQP and MQTT(Commonly used in IoT application) that can be used to build more large scale systems, however these have a steep learning curve and wont be ideal for demonstration purposes in this article.

Concepts

With this high level understanding of how the Event-Driven Architecture works, lets look and some of its core concepts. These will give you a platform to weigh the strengths of this architecture to others and possibly give you a basis on deciding when to apply it in your projects. We sha;; use the tool to demonstrate these concepts.

The Event broker

An Event Broker is a middleware platform that routes events from source to desired destination. This is accomplished using the publish-subscribe messaging pattern. In this pattern, independent system components connect to the event broker and exchange messages through the Event Broker.

Multiple event brokers Exist, some of the most popular ones include Apache Kafka, RabbitMQ, AWS EventBridge, and Redis. In this article, we will use Redis as our event broker. To follow along with the practical example atart you redis instance. Make sure its listening on port 6379 as thats where eda cli tool will try to access it from. Unfortunately I had to dard code the port value. You can run the following command to run Redis in a Docker Container.

docker run -d -p 6379:6379 redis

Event Portals

Event portals in Event-Driven Architecture are tools or platforms designed to facilitate the management, discovery, and governance of events within an event-driven system. They are especially valuable in modern distributed systems where events act as the primary means of communication between components.

You should not confuse them with Event Brokers as the definition might sound similar. Here are the differences

Aspect	Event Portal	Event Broker
Primary Function	Discovery, governance, and design of events	Real-time routing and delivery of events
Focus	Design-time and metadata management	Runtime event processing
Scope	Entire event-driven system	Specific event transportation tasks
Examples	AsyncAPI, Solace Event Portal	Kafka, RabbitMQ, AWS EventBridge

I found a good article on the Internet explaining Event Portals in detail. Read it here

Event Mesh

An event mesh is created and enabled through a network of interconnected event brokers. It’s a configurable and dynamic infrastructure layer for distributing events among decoupled applications, cloud services, and devices by dynamically routing events to any application, no matter where these applications are deployed in the world, in any cloud, on-premises, or IoT environment. Technically speaking, an event mesh is a network of interconnected event brokers that share consumer topic subscription information and route messages amongst themselves so they can be passed along to subscribers.

Topics in Event-Driven Architecture

In an event-driven system, topics are labels used to organize and route events. When an event is published, it is associated with a specific topic, and consumers subscribe to topics to receive events relevant to them. This approach allows producers to send events without needing to know which consumers will process them, while consumers only handle events tied to their subscribed topics. For example, an event with the topic orderCreated could be published by a producer, and any consumers subscribed to orderCreated would receive and process the event.

In this demonstration, we will set up a system with two components: the API server, which acts as the producer, and the consumer, which listens to a topic and logs the events it receives. First, you’ll need to start the API server by running the following command in your terminal:

eda api-server

This command initializes the server, which listens for POST requests. These requests include events that the server will publish to Redis, using the specified topic. Next, you’ll start a consumer that subscribes to a particular topic. To subscribe to the topic orderCreated, use the following command:

eda consumer --topic orderCreated

With both the API server and the consumer running, you can now send an event to the system. Use an HTTP client, such as curl or Postman, to send a POST request to the API server. For instance, sending the following request publishes an event to the topic orderCreated:

curl -X POST -H "Content-Type: application/json" \
-d '{"order_id": 123, "customer": "John Doe", "total": 49.99}' \
http://localhost:3000/topics/orderCreated

When this request is received, the API server sends the event to Redis, the event broker. Redis routes the event to all consumers subscribed to the topic orderCreated. If a consumer is actively listening, it will receive the event and log the message to its console. The consumer’s output will look like this:

Received event on topic: orderCreated
Event data: {"order_id": 123, "customer": "John Doe", "total": 49.99}

This demonstration shows how topics allow events to be routed to the appropriate consumers without requiring direct communication between the producer and the consumers. By associating events with topics, the system achieves flexibility and scalability while keeping its components decoupled.

Deferred Execution

Deferred execution is a key concept in event-driven architecture that differs significantly from traditional REST-based APIs. In a REST-based system, when you send a request, you typically wait for an immediate response. However, in event-driven systems, you don’t wait for a response when you publish an event. Instead, the event is handed off to an event broker, which holds or persists the event until consumers are ready to process it. This allows for more flexible and decoupled communication, as consumers can process events at their own pace, potentially much later.

In this approach, acting on one event might also cause new events to be generated and persisted in the event broker. These subsequent events can trigger further actions in other parts of the system, creating a chain of reactions, all while maintaining the flexibility of deferred processing.

To understand this concept better, let’s walk through a practical demonstration of deferred execution.

Practical Demonstration

In this demonstration, you will observe how events can be held in a queue and processed after a delay, demonstrating deferred execution. Here are the steps:

Start the API Server

First, start the API server, which allows you to publish events to the system:

eda api-server

This server listens for POST requests and routes event messages to the system.

Send an Event

Send a POST request to the /deferred-exec endpoint of the API server. The request body contains the event message you want to publish. For example:

curl -X POST -H "Content-Type: application/json" \
-d '{"task_id": 456, "description": "Process data files"}' \
http://localhost:3000/deferred-exec

When this request is sent, the API server pushes the event to Redis, the event broker, where it will be stored in a queue for processing.

Start the Master Node

Next, start the master node, which plays a key role in deferred execution by sending events from Redis to the event queue. To start the master node, run the following command:

eda consumer -d --master

Start a Consumer

Finally, run the consumer to process events from the event queue. The consumer checks the queue every 10 seconds, retrieves the events, and logs them to demonstrate deferred execution. To start the consumer, use the command:

eda consumer -d

Once the consumer is running, you will see a 10-second delay before the consumer processes events. For example, the consumer’s log might display:

Processing event: {"task_id": 456, "description": "Process data files"}

This delay illustrates how events can be stored in a queue and processed later, at the consumer’s convenience. Such deferred execution is particularly useful in scenarios where immediate processing isn’t required, allowing for greater flexibility and resource efficiency.

Eventual Consistency

Building on the concept of deferred execution, where events are processed at a later time rather than immediately, we encounter the idea of eventual consistency. In an event-driven system, since events are not processed instantly, there’s no guarantee that all components, such as databases or systems, are synchronized at any given moment. For example, if you have multiple systems—like a database, a master data management (MDM) system, and an ERP—they may not have the same state at the same time.

In our deferred execution demonstration, when an event (like a task or an order) is published to Redis, it may take time for the consumer to process it. Until the consumer acts on the event and updates the relevant systems, those systems might not reflect the latest state. However, while consistency isn’t immediate, we can rely on the system to eventually synchronize. This means that all stateful components will reach a consistent state over time, provided the system is functioning as intended and events are processed correctly.

For example, after publishing a task event with the API server, the consumer may process it after a delay, updating the system state accordingly. While the task's status may appear outdated immediately after publication, you can trust that the system will catch up and reflect the changes eventually.

This principle is particularly useful in distributed systems, where maintaining immediate consistency across all components can be impractical or inefficient. By embracing eventual consistency, we trade immediate synchronization for better scalability and resilience, trusting the system to converge to a consistent state in due time.

Choreography

In the world of event-driven architecture, choreography is a pattern where multiple components interact and coordinate their actions by responding to shared events. Unlike orchestration, where a central controller dictates the flow of operations, choreography allows each component to act independently based on the events it receives. This decentralized approach aligns well with the principles of EDA, promoting loose coupling and scalability.

Benefits of Choreography

Choreography ensures that components operate independently, reducing the risk of bottlenecks or single points of failure. In our system, the master node facilitates event flow but does not tightly couple itself to the consumers. This allows for easy scaling—new consumers can join without the master node needing to know about them, as long as they subscribe to the appropriate topics or queues.

By adopting choreography, you can build systems that are more resilient and flexible. Each component focuses on its responsibilities, reacting to events as needed, rather than relying on a central controller to orchestrate every action.

CQRS:Command Query Responsibility Segregation

CQRS is a method for scaling microservices by separating responsibilities. One service handles commands, such as updates or inserts, while another handles queries, such as fetching data. This separation is useful because query services often need to handle far more requests than commands, making it easier to scale the query service independently.

In an event-driven architecture, implementing CQRS is straightforward. Events in the system can be categorized by topics that include a specific action or verb, like "query" or "update." To scale the query service, you can simply deploy more instances of it and have them listen to the topics related to queries. This way, the system remains efficient, scalable, and easy to manage.

Conclusion

Event-Driven Architecture (EDA) is a powerful paradigm that enables systems to be more flexible, scalable, and resilient by decoupling components and allowing them to communicate through events. Throughout this article, we’ve explored the principles of EDA, discussed its core concepts like deferred execution, eventual consistency, choreography, and CQRS, and demonstrated how these concepts can be applied in practice using Redis and a custom-built CLI tool.

By embracing EDA, you can design systems that not only handle complex workflows efficiently but also adapt to changing demands with ease. Whether you're scaling query services, coordinating actions through events, or ensuring eventual consistency in distributed systems, EDA provides the tools and patterns to build robust architectures.

Thank you for taking the time to read this article! I hope it has been informative and inspires you to experiment with EDA in your own projects. About me, I am Beingana Jim Junior, a software engineering student at Makerere University, I love sharing insights from my learning and experiences. I’m passionate about software design and always eager to explore new technologies and solutions. Check out my Github and Portifolio website

Feel free to share your thoughts, questions, or feedback. I’d love to hear from you. Let’s continue building amazing systems together!

References

Event-Driven Architecture. Amazon Web Services. https://aws.amazon.com/event-driven-architecture/

Event-Driven Architecture (EDA) - A Complete Introduction. Confluent. https://www.confluent.io/learn/event-driven-architecture/

The Complete Guide to Event-Driven Architecture. Medium. https://medium.com/@seetharamugn/the-complete-guide-to-event-driven-architecture-b25226594227

My gift to 2024

Beingana Jim Junior — Sat, 28 Dec 2024 17:18:14 +0000

Building a Kubernetes Operator | A Practical Guide

Beingana Jim Junior ・ Dec 28 '24

#kubernetes #devops #go #tutorial

Building a Kubernetes Operator | A Practical Guide

Beingana Jim Junior — Sat, 28 Dec 2024 17:17:17 +0000

One of those great revolutionary technologies that have transformed how developers think of and Interact with cloud infrastructure is Kubernetes. Initially Developed at Google, Kubernetes, also known as K8s, is an open source system for automating deployment, scaling, and management of containerized applications. It is Designed on the same principles that allow Google to run billions of containers a week, Kubernetes can scale without increasing your operations team.

Despite its remarkable capabilities, Kubernetes is fundamentally a container orchestration technology. While it greatly simplifies deployment and scaling, it doesn't address all challenges that software engineers encounter in software development and DevOps. To Solve this Kubernetes provides ways it can be Extended and customized to meet your team's needs. It Provides Client Libraries in many languages. But even better are Kubernetes Operators.

A formal definition of a kubernetes operator can be:

A Kubernetes Operator is a method of automating the management of complex applications on Kubernetes. It extends Kubernetes' capabilities by using custom resources and controllers to manage the lifecycle of an application, automating tasks such as deployment, scaling, and updates. Operators encode the operational knowledge of an application into Kubernetes, making it easier to manage stateful or complex workloads with less manual intervention.

In this article. We shall go through a guide to get started building your own Custom kubernetes Operator. We shall cover different topics like Custom Resource Definitions, Controllers and look at the Kubernetes Controller Runtime.

Prerequisites

There are a few thing we need to know and have before continuing with this tutorial

A good understanding of Kubernetes and how to use it.
Programming Knowledge in the Go Programming Language.
Access to a Kubernetes Cluster (You can try it locally using Minikube or Kind)

To set up your environment you will first need to have Go installed. The Kubernetes Golang Client usually requires a specific Go version so depending on your time of reading this article it might have changed but for now i will use go1.21.6. To know what version you are using, run

go version
# Example output: go version go1.21.6 linux/amd64

Next you will need to have access to a Kubernetes cluster but for development purposes I would advise you use a local cluster from tools like Minikube or Kind. You can visit their websites for the installation steps.

Before we dive right into code. There are certain key concepts you will need to know so let's look at those first.

Concepts

Custom Resource Definitions (CRDS)

To understand CRDs you need to first know what a resource is. Pods, Deployments, Services etc are all resources. Formally

A resource is an endpoint in the Kubernetes API that stores a collection of API objects of a certain kind; for example, the built-in pods resource contains a collection of Pod objects.

Resources are in built within the Kubernetes API. But in our case, we said one of the main resons we build operators is to handle custom problems that Kubernetes does not solve out of the box. In some cases we might need to define our own resource objects. Forexample. Imagine we are building an operator that manages postgreSql databases, we would like to provide an API to define configurations of each database we initiate, we can do this by defining a Custom resource definition PGDatabase as an Object that stores the configuration of the database.

Example of a Custom Resource Definition for demostration

apiVersion: example.com/v1 # Every resource must have an API Version 
kind: PGDatabase # CRD Name
metadata:
  name: mydb
spec:
  config:
    port: 5432
    user: root
    dbname: mydb

  volumes:
  - pgvolume: /pgdata/

  envFrom: wordpress-secrets

Note: CRDs are just definitions of the actually objects or they just represent an actual object in the kubernetes cluster but are not the actual object. For example when you write a yaml file defining how a pod should be scheduled, that yaml just defines but its not the actual pod.

Therefore we can define a CRD as an extension of the Kubernetes API that is not necessarily available in a default Kubernetes installation. It represents a customization of a particular Kubernetes installation. Actually, many core Kubernetes functions are now built using custom resources, making Kubernetes more modular.

Controllers

Next Lets look at Kubernetes Controllers. As we have seen above, CRDs represent objects or state of objects in the Kubernetes Cluster, but we need something to reconcile or transform that state into actual resources on the cluster, this is where Controllers come in.

A Kubernetes controller is a software component within Kubernetes that continuously monitors the state of cluster resources and takes action to reconcile the actual state of the cluster with the desired state specified in resource configurations (YAML files).

Controllers are responsible for managing the lifecycle of Kubernetes objects, such as Pods, Deployments, and Services. Each controller typically handles one or more resource types and performs tasks like creating, updating, or deleting resources based on a declared specification.
Here's a breakdown of how a Kubernetes controller operates:

Observe: The controller watches the API server for changes to resources it's responsible for. It monitors these resources by querying the Kubernetes API periodically or via an event stream.
Analyze: It compares the actual state (current conditions of the resources) to the desired state (specified in configurations).
Act: If there’s a discrepancy between the actual and desired state, the controller performs operations to align the actual state with the desired state. For example, a Deployment controller might create or terminate Pods to match the specified replica count.
Loop: The controller operates in a loop, continuously monitoring and responding to changes to ensure the system’s resources are always in the desired state.

Controller Runtime

Kubernetes provides a set of tools to build native Controllers and these are Known as the Controller Runtime.

Lets take a deep look into the Controller runtime because its what we shall be using to build out Operators

A Deep look into the Controller Runtime

Controller Runtime is a set of libraries and tools in the Kubernetes ecosystem, designed to simplify the process of building and managing Kubernetes controllers and operators. It’s part of the Kubernetes Operator SDK and is widely used to develop custom controllers that can manage Kubernetes resources, including custom resources (CRDs).

Controller Runtime provides a structured framework for controller logic, handling many of the lower-level details of working with the Kubernetes API, so developers can focus more on defining the behavior of the controller and less on boilerplate code. It is written in Go and builds on the Kubernetes client-go libraries.

To use it, you can add it to you golang project by importing it like this:

package controller

// Controller runtime packages
import (
    "sigs.k8s.io/controller-runtime"
    "sigs.k8s.io/controller-runtime/pkg/client"
)

Note: The Controller runtime is not the only way one can build a Kubernetes Operator, there are multiple ways to do so such as using the Operator Framework SDK or Kubebuilder, which are both frameworks built on top of the Controller runtime and utilize it under the hood to assist you when building complex Operators. You could even build an application that utilizes the Kubernetes Rest API through client libraries in various languages such as Python, Java, JavaScript etc depending on your tech stack. Find the Full list of Client Libraries on the Kubernetes Documentation.
In this article, we will use the Controller runtime because it offers flexibility and provides a hands-on understanding of how Controllers work internally. This approach is ideal for gaining deeper insight into the inner workings of Kubernetes Operators while maintaining the ability to extend or customize as needed.

Key Components of Controller Runtime

The Controller Runtime has several key components that streamline the process of building and running Kubernetes controllers. Together, these components create a robust framework for building Kubernetes controllers.

The Manager initiates and manages other components; the Controller defines reconciliation logic; the Client simplifies API interactions; the Cache optimizes resource access; Event Sources and Watches enable event-driven behavior; and the Reconcile Loop ensures continuous alignment with the desired state. These components make it easier to build controllers and operators that efficiently manage Kubernetes resources, allowing for custom automation and orchestration at scale.

1. Manager

The Manager is the main entry point of a controller or operator, responsible for initializing and managing the lifecycle of other components, such as controllers, caches, and clients. Developers usually start by creating a Manager in the main function, which acts as the foundational setup for the entire operator or controller.

It provides shared dependencies (e.g., the Kubernetes client and cache) that can be reused across controllers, allowing multiple controllers to be bundled within a single process.

The Manager also coordinates starting and stopping all controllers it manages, ensuring that they shut down gracefully if the Manager itself is stopped.

2. Controller

The Controller is the core component that defines the reconciliation logic responsible for adjusting the state of Kubernetes resources. It is a control loop that monitors a cluster's state and makes changes to move it closer to the desired state

Each controller watches specific resources, whether built-in (e.g., Pods, Deployments) or custom resources (CRDs). It includes a Reconcile function that’s triggered whenever a resource changes, allowing the controller to bring the current state in line with the desired state.

Developers specify which resources the controller should watch, and Controller Runtime will automatically track and respond to events (like create, update, delete) for those resources. In a controller for managing custom Foo resources, the Reconcile function might create or delete associated resources based on Foo specifications.

2. Client

When building a Kubernetes operator, you need some interface to interact with the Kubernetes Cluster and carry out your operations. Just like kubectl the command line client we use, the Controller runtime provided a client in its SDK tools. This client is also used to interact with the Kubernetes API pragrammatically in your code.

The Client is an abstraction that simplifies interactions with the Kubernetes API, enabling CRUD operations on resources.

This component allows for easy creation, reading, updating, and deletion of Kubernetes resources.
The Client is integrated with the cache to provide efficient access to resources without overloading the API server.

Controller Runtime’s Client extends the basic Kubernetes client-go library, making API calls more straightforward by handling details like retries and caching behind the scenes.

Using the Client, developers can create a Pod directly from the controller logic with a single line of code, client.Create(ctx, pod), without having to worry about raw API requests.

3. Event Sources and Watches

Event Sources and Watches determine the resources that a controller will monitor for changes, providing a way to respond to specific events in the cluster.

Event Sources define what triggers the controller’s reconciliation loop, which can be based on changes in specific Kubernetes resources.
Watches monitor these resources, allowing the controller to act on create, update, or delete events as needed.

Developers can define multiple Watches for a single controller, which is useful if the controller’s behavior depends on multiple resources.

A controller managing a custom App resource might watch Pods, Services, and ConfigMaps, reacting to changes in any of these resources by adjusting the App accordingly.

4. Reconcile Loop

The Reconcile loop is the heart of the controller, implementing the main logic that determines the steps to bring resources into the desired state.
Each controller’s Reconcile function checks the actual state of a resource and then applies necessary changes to make it conform to the desired state.
This loop continues indefinitely, with each reconciliation acting as a self-healing mechanism to correct any divergence from the specification.
The Reconcile loop is usually idempotent, meaning it can be repeated without causing unintended side effects, ensuring consistency even with frequent updates.
In a Reconcile function, the controller might find that a Deployment lacks the specified number of replicas, so it updates the Deployment configuration to match the desired replica count.

A Practical Example

Now that you have grasped a few required concepts, lets take a practical approach by building a simple kubernetes oparator and deploying it on our Kubernetes Cluster. This will help you to apply the concepts above and understand the process more.

In this section, we will build a simple Kubernetes operator that helps deploy applications using a custom resource definition (CRD). This operator will automate the creation of Deployments and Services based on the application configuration provided in the custom resource. By the end, you'll have a working operator deployed on a Kubernetes cluster and understand its key components.

All code will be made available on Github at https://github.com/jim-junior/crane-operator

I will ensure code is commented in detail to ensure you can easily read and understand it

Operator Requirements

Our goal is to minimize the complexity of defining Kubernetes resources for deploying an application. Instead of writing multiple YAML files for Deployments, Services, etc., we will define a single CRD Application that encapsulates all required configurations. The operator will then create the necessary Kubernetes resources automatically.

Requirements for the Example:

The operator shall define a CRD Application that includes all the configuration of the application.
From the CRD, it should create the associated Kubernetes resources such and Deployments, Services
We shall Implement a controller to reconcile the state of Application resources.
Use the controller to create and manage Deployments and Services.

Now that we have a high level understanding of what our Operator is supposed to do and accomplish, lets setup our project and get started building the operator.

Setting up the Project

As mentioned earlier, we will use the Go programming language for this tutorial. Why Go? Primarily because the Controller runtime, as well as Kubernetes itself, is built using Go. Additionally, Go is specifically designed with features that make it ideal for building cloud-native applications. Its architecture ensures that your Operator will be well-suited to scale efficiently in cloud environments. We are using go1.21.6

So lets begin by initialising out Go project:

mkdir app-operator && cd app-operator

go mod init github.com/jim-junior/crane-operator

We shall then install the go dependencies that we shall use in our project. You can install then by running this in the command line.

go get sigs.k8s.io/controller-runtime
go get k8s.io/apimachinery
go get k8s.io/client-go

Next lets setup our project file/directory structure. Depending on your development paradigm you can choose one that favours you but for now we shall use this simple that i have found to work in most Operator projects.

app-operator/
├── api/                  # Contains CRD definitions
├── cmd/controller/         
    ├──reconciler.go      # Contains the reconciliation logic
├── main.go               # Entry point for the operator
├── config/               # Configuration files for testing and deployment

Defining the Custom Resource Definition (CRD)

As metioned above, the CRD defines the specification of how our Object will look like. From familiarity with deploying multiple applications or web services, there are a few things an application needs:

A port to expose your application
A Volume, incase you want to Persist Data
A Container Image for application distribution and deployment
Enviroment Variables

Since this is just an example we shall keep the requirements as minimal as possible, its not like we are building some operator to be used in an actual Organisation. Plus I dont want the code to become huge.

When defining a Custom Resource Definition (CRD), it must be specified in two formats. The first is as a yaml or json OpenAPI specification (yaml is preffered is its more human readable), which can be applied using kubectl apply to install the CRD on a Kubernetes cluster. The second is as a Go language specification, used in your code to define and interact with the CRD programmatically.

While tools like Operator SDK and Kubebuilder can automatically generate one or both of these formats for you, it's important for a developer building a Kubernetes Operator to understand the configurations being generated. This knowledge is invaluable for debugging or handling custom scenarios that may arise.

Lets begin first with an example showing how we would like our CRD to look like. This will help use comeup with a Specification. Incase you dont understand some aspects of it, dont worry, we shall look into everything.

apiVersion: operator.com/v1
kind: Application
metadata:
  name: mysql
spec:
  image: mysql:9.0
  ports:
  - internal: 3306
    external: 3306

  volumes:
  - volume-name: /data/

  envFrom: mysql-secrets

Before we define the Open API Specification, lets look at what each component means in out CRD above.

apiVersion: operator.com/v1

This defines the Version of your resource. Every resource on kubernetes MUST have an api version. Even inbuilt kubernetes resources have. I would recommend versioning your resources while following the Kubernetes API versioning convetions.

Kubernetes API versioning follows conventions that reflect the maturity and stability of an API. Here’s a list of the common versioning conventions and what each represents:

Alpha (e.g., v1alpha1): Experimental and not stable.
Beta (e.g., v1beta1): More stable than alpha but still under active development.
Stable (e.g., v1): Fully stable and backward-compatible.

Kubernetes APIs use a convention of <group>/<version> (e.g., apps/v1), where:

Group: The API group (e.g., apps, batch, core).
Version: The maturity level of the API (v1alpha1, v1beta1, v1).

Custom Resource Definitions (CRDs) follow the same versioning principles as core Kubernetes APIs.

Lets move on to.

kind: Application

In Kubernetes, the kind field in a resource manifest specifies the type of resource being defined or manipulated. It is a crucial identifier that tells Kubernetes which resource object the YAML or JSON file represents, enabling the API server to process it accordingly. Learn more here

Note: The value is case-sensitive and typically written in PascalCase (e.g., ConfigMap, Deployment).

Next is

spec:
  .....

This is where you define the properties of your CRD.

Open API Specification

We can now define the Open API Specification for the CRD. You basically have to transform the above yaml into an Open API Specification. You can learn more about Open API Specification on its Documentation. But its pretty straight forward. For the above CRD this is what it would look like.

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: applications.operator.com
spec:
  group: operator.com
  names:
    kind: Application
    plural: applications
    singular: application
  scope: Namespaced
  versions:
  - name: v1
    served: true
    storage: true
    schema:
      openAPIV3Schema:
        type: object
        properties:
          apiVersion:
            type: string
          kind:
            description: 'You can provide a description'
            type: string
          metadata:
            type: object
          spec:
            type: object
            properties:
              # image name
              image:
                type: string
              # Volumes
              volumes:
                type: array
                items:
                  type: object
                  properties:
                    volume-name:
                      type: string
                    path:
                      type: string
              # Port Configuration
              ports:
                type: array
                items:
                  type: object
                  properties:
                    name:
                      type: string
                    internal:
                      type: integer
                      format: int64
                    external:
                      type: integer
                      format: int64
              # Environment variables
              envFrom:
                type: string

Note that even the Specification is also a resource in Kubernetes of kind CustomResourceDefinition

You can now store that code in the file located in app-operator/config/crd.yml.

Go Lang CRD Definition

We can now define our CRD in Go code. When defininga CRD in Go. We need to do the following

Define the CRD Specification
Define a deepCopy function that defines how kubernetes copies the CRD object into another object
Setup code for Registering the CRD

We use Go Structs to define the CRD. Probably due to how it is easy to define JSON-like data structures usung structs. Create a file in app-operator/api/v1/application.go and save the following code in it.

package v1

import metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

// This defines an instance of multiple Application resources
type ApplicationList struct {
  metav1.TypeMeta `json:",inline"`
  metav1.ListMeta `json:"metadata,omitempty"`

  Items []Application `json:"items"`
}

// This defines our CRD
type Application struct {
  metav1.TypeMeta   `json:",inline"`
  metav1.ObjectMeta `json:"metadata,omitempty"`

  Spec ApplicationSpec `json:"spec"`
}

type ApplicationSpec struct {
  Image     string               `json:"image"`
  Volumes   []ApplicationVolume  `json:"volumes"`
  Ports     []ApplicationPortMap `json:"ports"`
  EnvFrom   string               `json:"envFrom"`
}

type ApplicationVolume struct {
  VolumeName string `json:"volume-name"`
  Path       string `json:"path"`
}

type ApplicationPortMap struct {
  Name     string `json:"name"`
  Internal int    `json:"internal"`
  External int    `json:"external"`
}

Next lets write the code that defines the Deep Copy functions. In you project create a file in app-operator/api/v1/deepcopy.go. And add the following code

package v1

import "k8s.io/apimachinery/pkg/runtime"

// DeepCopyInto copies all properties of this object into another object of the
// same type that is provided as a pointer.
func (in *Application) DeepCopyInto(out *Application) {
  out.TypeMeta = in.TypeMeta
  out.ObjectMeta = in.ObjectMeta
  out.Spec = ApplicationSpec{
    Volumes: in.Spec.Volumes,
    Ports:   in.Spec.Ports,
    EnvFrom: in.Spec.EnvFrom,
    Image:   in.Spec.Image,
  }
}

// DeepCopyObject returns a generically typed copy of an object
func (in *Application) DeepCopyObject() runtime.Object {
  out := Application{}
  in.DeepCopyInto(&out)

  return &out
}

// DeepCopyObject returns a generically typed copy of an object
func (in *ApplicationList) DeepCopyObject() runtime.Object {
  out := ApplicationList{}
  out.TypeMeta = in.TypeMeta
  out.ListMeta = in.ListMeta

  if in.Items != nil {
    out.Items = make([]Application, len(in.Items))
    for i := range in.Items {
      in.Items[i].DeepCopyInto(&out.Items[i])
    }
  }

  return &out
}

Lastly lets write the code that defines how our CRD is registed. In you project create a file in app-operator/api/v1/register.go. And add the following code

package v1

import (
  metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
  "k8s.io/apimachinery/pkg/runtime"
  "k8s.io/apimachinery/pkg/runtime/schema"
)

// Define the API group name for the custom resource
const GroupName = "operator.com"
// Define the API group version for the custom resource
const GroupVersion = "v1"

// Create a GroupVersion object that combines the group and version
var SchemeGroupVersion = schema.GroupVersion{Group: GroupName, Version: GroupVersion}


// SchemeBuilder is a runtime.SchemeBuilder used to add types to the scheme
var (
  SchemeBuilder = runtime.NewSchemeBuilder(addKnownTypes)  // Initializes the SchemeBuilder with the addKnownTypes function
  AddToScheme   = SchemeBuilder.AddToScheme  // Provides a shorthand for adding types to the scheme
)


// addKnownTypes registers the custom resource types with the runtime.Scheme
func addKnownTypes(scheme *runtime.Scheme) error {
  // Register the custom resources Application and ApplicationList with the scheme
  scheme.AddKnownTypes(SchemeGroupVersion,
    &Application{},
    &ApplicationList{},
  )

  // Add the group version to the scheme for metav1 objects
  metav1.AddToGroupVersion(scheme, SchemeGroupVersion)
  return nil // Return nil to indicate success
}

We can now move on to the next section of Implementing the Controller that will transform our CRD state into the desired objects on the kubernetes cluster.

Implementing the Controller

Now that we have our CRD definitions implemented, lets move on to creating our controller that will watch the CRD state and transform it into desired Kubernetes onjects which will be Deployements and Services. To accomplish this we shall visits a few of the concepts we metioned earlier about the Controllers. We shall create the following:

A Manager that will be the entry point of our Controller
The Reconcile function that reconciles the CRD state into desired state on the Cluster
Utility functions that carry out the tasks our operator intends to accomplish

In the app-operator/cmd/controller/reconciler.go, paste this code. Dont wory we shall look at each block in detail and what it accomplishes.

package controller

import (
  "context"
  "errors"
  "fmt"
  "os"
  "path/filepath"

  cranev1 "github.com/jim-junior/crane-operator/api/v1"
  craneKubeUtils "github.com/jim-junior/crane-operator/kube"

  k8serrors "k8s.io/apimachinery/pkg/api/errors"
  "k8s.io/apimachinery/pkg/runtime"
  utilruntime "k8s.io/apimachinery/pkg/util/runtime"
  "k8s.io/client-go/kubernetes"
  "k8s.io/client-go/rest"
  "k8s.io/client-go/tools/clientcmd"
  "k8s.io/client-go/util/homedir"
  ctrl "sigs.k8s.io/controller-runtime"
  "sigs.k8s.io/controller-runtime/pkg/client"
  "sigs.k8s.io/controller-runtime/pkg/log"
  "sigs.k8s.io/controller-runtime/pkg/log/zap"
)

// Global variables for the Kubernetes scheme and logger
var (
  scheme   = runtime.NewScheme()
  setupLog = ctrl.Log.WithName("setup")
)

// Initialize the scheme by registering the cranev1 API group
func init() {
  utilruntime.Must(cranev1.AddToScheme(scheme))
}

// Reconciler structure to handle reconciliation logic
type Reconciler struct {
  client.Client            // Controller-runtime client
  scheme     *runtime.Scheme // Scheme for managing API types
  kubeClient *kubernetes.Clientset // Kubernetes clientset for direct API calls
}

// Reconcile function handles the main logic for the controller
func (r *Reconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
  log := log.FromContext(ctx).WithValues("application", req.NamespacedName) // Create contextual logger
  log.Info("reconciling application")

  // Fetch the Application resource by name and namespace
  var application cranev1.Application
  err := r.Client.Get(ctx, req.NamespacedName, &application)
  if err != nil {
    // If resource is not found, attempt to clean up associated resources
    if k8serrors.IsNotFound(err) {
      err = craneKubeUtils.DeleteApplication(ctx, req, r.kubeClient)
      if err != nil {
        return ctrl.Result{}, fmt.Errorf("couldn't delete resources: %s", err)
      }
      return ctrl.Result{}, nil
    }
  }

  // Create or update the Kubernetes deployment for the Application resource
  err = craneKubeUtils.ApplyApplication(ctx, req, application, r.kubeClient)
  if err != nil {
    return ctrl.Result{}, fmt.Errorf("couldn't create or update deployment: %s", err)
  }

  return ctrl.Result{}, nil // Reconcile completed successfully
}

// RunController initializes and starts the Kubernetes controller
func RunController() {
  var (
    config *rest.Config
    err    error
  )

  // Determine the kubeconfig file path (used for local development)
  kubeconfigFilePath := filepath.Join(homedir.HomeDir(), ".kube", "config")
  if _, err := os.Stat(kubeconfigFilePath); errors.Is(err, os.ErrNotExist) {
    // If kubeconfig does not exist, try to use in-cluster configuration
    config, err = rest.InClusterConfig()
    if err != nil {
      panic(err.Error()) // Exit if no valid configuration is found
    }
  } else {
    // Load configuration from kubeconfig file
    config, err = clientcmd.BuildConfigFromFlags("", kubeconfigFilePath)
    if err != nil {
      panic(err.Error())
    }
  }

  // Create a Kubernetes clientset
  clientset, err := kubernetes.NewForConfig(config)
  if err != nil {
    panic(err.Error())
  }

  // Set up the logger for the controller
  ctrl.SetLogger(zap.New())

  // Create a new manager for the controller
  mgr, err := ctrl.NewManager(config, ctrl.Options{
    Scheme: scheme,
  })
  if err != nil {
    setupLog.Error(err, "unable to start manager")
    os.Exit(1)
  }

  // Create and register the reconciler with the manager
  err = ctrl.NewControllerManagedBy(mgr).
    For(&cranev1.Application{}). // Specify the resource type the controller manages
    Complete(&Reconciler{
      Client:     mgr.GetClient(),   // Use manager's client
      scheme:     mgr.GetScheme(),  // Use manager's scheme
      kubeClient: clientset,        // Use clientset for direct API calls
    })

  if err != nil {
    setupLog.Error(err, "unable to create controller")
    os.Exit(1)
  }

  // Start the manager and handle graceful shutdown
  setupLog.Info("starting manager")
  if err := mgr.Start(ctrl.SetupSignalHandler()); err != nil {
    setupLog.Error(err, "error running manager")
    os.Exit(1)
  }
}

Lets explain the Controller Code

Lets take a deep dive into the code above so we can have a good understanding of whats going on in that file.

At the top of the file we are importing the required libraries that we shall use. Read the comments in this code snippet to understand what each import does.

import (
  "context" // Provides functionality for managing and passing context, especially useful in request-scoped operations.
  "errors"  // Standard Go package for creating and handling errors.
  "fmt"     // Provides formatted I/O with functions similar to C's printf and scanf.
  "os"      // Handles OS-level functionalities such as reading environment variables and file system operations.
  "path/filepath" // Helps in manipulating and building file paths in a cross-platform way.

  cranev1 "github.com/jim-junior/crane-operator/api/v1" // Imports the custom CRD definitions (e.g., Application) for this operator.
  craneKubeUtils "github.com/jim-junior/crane-operator/kube" // Imports helper utilities for interacting with Kubernetes resources.

  k8serrors "k8s.io/apimachinery/pkg/api/errors" // Provides utilities for working with Kubernetes API errors.
  "k8s.io/apimachinery/pkg/runtime" // Handles runtime types and schemes for Kubernetes objects.
  utilruntime "k8s.io/apimachinery/pkg/util/runtime" // Contains utility functions for runtime error handling and recovery.
  "k8s.io/client-go/kubernetes" // Kubernetes client-go library for interacting with the Kubernetes API server.
  "k8s.io/client-go/rest" // Provides tools for working with REST configurations, especially for in-cluster access.
  "k8s.io/client-go/tools/clientcmd" // Handles loading and parsing kubeconfig files for out-of-cluster Kubernetes access.
  "k8s.io/client-go/util/homedir" // Utility package to get the user's home directory path.

  ctrl "sigs.k8s.io/controller-runtime" // Main package for building controllers using the Kubernetes Controller Runtime.
  "sigs.k8s.io/controller-runtime/pkg/client" // Provides a dynamic client for interacting with Kubernetes objects.
  "sigs.k8s.io/controller-runtime/pkg/log" // Utilities for logging within the Controller Runtime framework.
  "sigs.k8s.io/controller-runtime/pkg/log/zap" // Provides a Zap-based logger for the Controller Runtime.
)

We have an init function that initialize the Kubernetes scheme by registering the API group we defined for our CRD.

Next lets look ar the RunController function at the bottom of the file. First we need to get access to a Kuberentes client or clientset that will allow use to interact with the Kubernetes API Server when carrying out CRUD operations on our cluster. That what these first 27 lines are trying to accomplish. We shall call the RunController function in the main.go file of our program as it will be the entry point of out Kubernetes Operator

kubeconfigFilePath := filepath.Join(homedir.HomeDir(), ".kube", "config")
  if _, err := os.Stat(kubeconfigFilePath); errors.Is(err, os.ErrNotExist) {
    // If kubeconfig does not exist, try to use in-cluster configuration
    config, err = rest.InClusterConfig()
    if err != nil {
      panic(err.Error()) // Exit if no valid configuration is found
    }
  } else {
    // Load configuration from kubeconfig file
    config, err = clientcmd.BuildConfigFromFlags("", kubeconfigFilePath)
    if err != nil {
      panic(err.Error())
    }
  }

  // Create a Kubernetes clientset
  clientset, err := kubernetes.NewForConfig(config)
  if err != nil {
    panic(err.Error())
  }

We then setup logging wth zap. This will help us in debugging

ctrl.SetLogger(zap.New())

We then setup our Manager.

// Create a new manager for the controller
  mgr, err := ctrl.NewManager(config, ctrl.Options{
    Scheme: scheme,
  })
  if err != nil {
    setupLog.Error(err, "unable to start manager")
    os.Exit(1)
  }

  // Create and register the reconciler with the manager
  err = ctrl.NewControllerManagedBy(mgr).
    For(&cranev1.Application{}). // Specify the resource type the controller manages
    Complete(&Reconciler{
      Client:     mgr.GetClient(),   // Use manager's client
      scheme:     mgr.GetScheme(),  // Use manager's scheme
      kubeClient: clientset,        // Use clientset for direct API calls
    })

  if err != nil {
    setupLog.Error(err, "unable to create controller")
    os.Exit(1)
  }

  // Start the manager and handle graceful shutdown
  setupLog.Info("starting manager")
  if err := mgr.Start(ctrl.SetupSignalHandler()); err != nil {
    setupLog.Error(err, "error running manager")
    os.Exit(1)
  }

This code initializes and starts the core components of the Kubernetes controller. First, a manager (mgr) is created using the ctrl.NewManager function, which serves as the runtime environment for the controller, handling shared resources, clients, and the scheme that defines the resource types the manager can work with. If the manager cannot start due to configuration issues, an error is logged, and the program exits. Next, a reconciler is created and registered with the manager. The reconciler defines the logic for ensuring the desired state of the custom resource (Application) matches the actual state in the cluster. This is done using ctrl.NewControllerManagedBy, which specifies the resource type the controller will manage and configures the reconciler with the manager's client, scheme, and a Kubernetes clientset for making direct API calls. If the controller cannot be created or registered, the program exits with an error. Finally, the manager is started using mgr.Start, which begins watching the specified resource and handling reconciliation requests. A signal handler is set up to ensure graceful shutdowns. If the manager fails to start, an error is logged, and the program exits. This setup combines the manager and reconciler to enable the operator to monitor and maintain the desired state of custom resources in the Kubernetes cluster.

Next lets define our Reconciler and Reconcile function.

We define the Reconciler struct. This is the struct that we used in the ctrl.NewControllerManagedBy when we initialized the controller. The struct should have a method called Reconcile that handles the main logic for the controller, i.e it reconciles the desired state into actual Kubernetes Objects.

type Reconciler struct {
  client.Client
  scheme     *runtime.Scheme
  kubeClient *kubernetes.Clientset
}

func (r *Reconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
  log := log.FromContext(ctx).WithValues("application", req.NamespacedName) // Create contextual logger
  log.Info("reconciling application")

  // Fetch the Application resource by name and namespace
  var application cranev1.Application
  err := r.Client.Get(ctx, req.NamespacedName, &application)
  if err != nil {
    // If resource is not found, attempt to clean up associated resources
    if k8serrors.IsNotFound(err) {
      err = craneKubeUtils.DeleteApplication(ctx, req, r.kubeClient)
      if err != nil {
        return ctrl.Result{}, fmt.Errorf("couldn't delete resources: %s", err)
      }
      return ctrl.Result{}, nil
    }
  }

  // Create or update the Kubernetes deployment for the Application resource
  err = craneKubeUtils.ApplyApplication(ctx, req, application, r.kubeClient)
  if err != nil {
    return ctrl.Result{}, fmt.Errorf("couldn't create or update deployment: %s", err)
  }

  return ctrl.Result{}, nil // Reconcile completed successfully
}

Note: I will not include utility functions imported as craneKubeUtils because they are not really necessary for this article but they are basically functions that create Deployements and Services from the CRD Spec. However, in the code hosted on the GitHub repository, you can find them in this file. https://github.com/jim-junior/crane-operator/blob/main/kube/application.go

Next in your main.go file import the RunController function from the controller package and call it in you main() function.

package main

import "github.com/jim-junior/crane-operator/cmd/controller"

func main() {
  controller.RunController()
}

Testing the Controller

Now that we are done writing the code for our controller, we can test it. First you will have to have you kubernetes cluster running and you config setup. Make sure the host machine that you are running it on is already connected to the kubernetes cluster. you can verify this by running any kubectl command such as:

kubectl get nodes
# If it setup it might return something like this
# NAME       STATUS   ROLES           AGE    VERSION
# minikube   Ready    control-plane   228d   v1.27.4

# If its not setup it might return an error like: 
# E1222 11:35:37.597805   25720 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp [::1]:8080: connectex: No connection could be made because the target machine actively refused it.

If all is well you can move on to test our controller by running:

go run main.go

The expected log output is something like this

{"level":"info","ts":"2024-12-22T11:08:23+03:00","logger":"setup","msg":"starting manager"}
{"level":"info","ts":"2024-12-22T11:08:23+03:00","logger":"controller-runtime.metrics","msg":"Starting metrics server"}
{"level":"info","ts":"2024-12-22T11:08:23+03:00","msg":"Starting EventSource","controller":"application","controllerGroup":"cloud.cranom.tech","controllerKind":"Application","source":"kind source: *v1.Application"}
{"level":"info","ts":"2024-12-22T11:08:23+03:00","msg":"Starting Controller","controller":"application","controllerGroup":"cloud.cranom.tech","controllerKind":"Application"}
{"level":"info","ts":"2024-12-22T11:08:23+03:00","logger":"controller-runtime.metrics","msg":"Serving metrics server","bindAddress":":8080","secure":false}
{"level":"info","ts":"2024-12-22T11:08:23+03:00","msg":"Starting workers","controller":"application","controllerGroup":"cloud.cranom.tech","controllerKind":"Application","worker count":1}

If your getting an error or having a hard time running the controller for some reason, you can open an issue on the Github Repository where the source code is hosted or you can leave it in the comment section if your reading this article on Dev.to or Medium.

If all is good. We can now try applying an example CRD and test if our operator is carrying out its expected functionality.

In the yaml/examples directory of the github repository i have placed there a cofigurations or our custom Application resource. These configuration are for deploying a Wordpress instance with a Mysql database. there are three files. mysql.yaml, wp-secrets.yml and wordpress.yml.

You can apply them by running tthe kubectl apply commands in this order.

# Secrets for the Environment Variables
kubectl apply yaml/examples/wp-secrets.yml
# Mysql instance
kubectl apply yaml/examples/mysql.yml
# Wordpress instance
kubectl apply yaml/examples/wordpress.yml

You can also copy the code from these files on Github

The configurations in those files are for a Wordpress application that will listen on the node port of 30080.

Deploying the Operator

Since we now know the kubernetes operator works, Lets move on to deploying it. To deploy it, we shall have to carry out the following steps.

Generate a docker image for the operator
Publish it to the Docker registry
We can they deploy it on our cluster.

We shall move on to generating a container image for the Kubernetes Operator. We shall use the golang image as our base image for the container since this is a golang project. Copy this code and add it to your DockerFile in the root of you project.

# Building the binary of the App
FROM golang:1.22.5 AS build

# `boilerplate` should be replaced with your project name
WORKDIR /go/src/boilerplate

# Copy all the Code and stuff to compile everything
COPY . .

# Downloads all the dependencies in advance (could be left out, but it's more clear this way)
RUN go mod download

# Builds the application as a staticly linked one, to allow it to run on alpine
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -a -installsuffix cgo -o app .


# Moving the binary to the 'final Image' to make it smaller
FROM alpine:latest as release

WORKDIR /app


# `boilerplate` should be replaced here as well
COPY --from=build /go/src/boilerplate/app .

# Add packages
RUN apk -U upgrade \
  && apk add --no-cache dumb-init ca-certificates \
  && chmod +x /app/app


ENTRYPOINT [ "/app/app" ]

You can then build the container using docker and then publish it to a container registry of your choice.

Then next is to deploy our cluster. We shall create a Kubernetes deployment for our Operator. Below i have a simple Kubernetes Deployment object for the operator.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: crane-operator
spec:
  replicas: 1
  selector:
    matchLabels:
      app: crane-operator
  template:
    metadata:
      labels:
        app: crane-operator
    spec:
      containers:
      - name: controller
        image: jimjuniorb/crane-operator:latest

Congrats!!! you now have a running Kubernetes operator on your cluster. You can now adjust it in what ever way you would like.

I would recommend looking into frameworks like Operator SDK or Kube builder if you want to build more complex Operators. I have also included a Github Action workflow file for deploying the Operator using GitHub actions each time a new Release tag is created.

Well thats it for Today. Thanks for following through till the End of this article. You can check our my other articles on my Blog. Below are a few references i used that you might find helpful.

References

Brandon Philips, (November 3, 2016). Introducing Operators: Putting Operational Knowledge into Software. Internet Archive Wayback Machine. https://web.archive.org/web/20170129131616/https://coreos.com/blog/introducing-operators.html

CNCF TAG App-Delivery Operator Working Group, CNCF Operator White Paper - Final Version. Github. https://github.com/cncf/tag-app-delivery/blob/163962c4b1cd70d085107fc579e3e04c2e14d59c/operator-wg/whitepaper/Operator-WhitePaper_v1-0.md

Kubernetes Documentation, Custom Resources. https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/

Kubernetes Documentation, Controllers. https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/

Youtube, Kubernetes API Versioning: A Deep dive. https://www.youtube.com/live/-jtGM6WnF1Q?si=-6tfrlwyTf-NSizL

How to create a responsive custom video player in React

Beingana Jim Junior — Sat, 04 May 2024 13:10:35 +0000

Video players are essential for applications that showcase video content. While the default HTML5 player offers basic functionalities like play, pause, and volume control, its appearance can vary significantly across browsers and devices. This inconsistency can be a drawback for a polished user experience.

In this article, we'll guide you through creating a responsive custom video player using a library I built called reactjs-media. This approach lets you avoid the limitations of the default HTML5 player and the complexity of building a video player from scratch. The reactjs-media library provides more react approach to building your custom player with a wide range of features and customization options.

UPDATE: The reactjs-media documentation was shifted to open.cranom.tech

Getting started

Before we dive into building our custom video player, I assume your already have a React project set up. If not, you can create a new React project using Create React App or Vite or any of your preferred React setup.

We shall then go ahead and install the reactjs-media library using npm or yarn.

npm install reactjs-media

yarn add reactjs-media

Creating a custom video player

To create a Video player we shall start with the default player provided by the library(Which is Awesome too).

All you have to do is import the Video component from reactjs-media

import { Video } from "reactjs-media";

const App = () => {
  return (
    <div>
      <Video
        src={"/video.mkv"}
        controls={true}
        height={500}
        width={800}
        poster={
          "https://hips.hearstapps.com/hmg-prod/images/ripley-pa-108-011822-01629-r-661067043d66f.jpg?resize=980:*"
        }
      />
    </div>
  );
};

And Voila! You have a video player in your application.

What are the props?

The Video Component accepts a number props that are defined under the VideoProps interface(If you use TypeScript in your Projects). These can be both attributes of the player or event handlers.

We shall start with the basics:

src: The source of the video file.
controls: A boolean value to show or hide the video controls. It defaults to false
height: The height of the video player.(This is added as maxHeight in the styles)
width: The width of the video player.(This is added as maxWidth in the styles)
poster: The poster image to be displayed before the video starts playing.
contextMenu: A boolean value to show or hide the context menu. It defaults to false

Handling Events

The Video Component also provides a number of event handlers that you can use to listen to events on the video player.

onPlay: This event is fired when the video starts playing.
onPause: This event is fired when the video is paused.
onEnded: This event is fired when the video has ended.
onTimeUpdate: This event is fired when the video is playing and the time is updated.
onVolumeChange: This event is fired when the volume of the video is changed.
onSeeking: This event is fired when the video is seeking.
onSeeked: This event is fired when the video has finished seeking.
onCanPlay: This event is fired when the video can be played.

Example:

<Video
  src={"/video.mkv"}
  controls={true}
  height={500}
  width={800}
  poster={
    "https://hips.hearstapps.com/hmg-prod/images/ripley-pa-108-011822-01629-r-661067043d66f.jpg?resize=980:*"
  }
  onPlay={() => console.log("Video is playing")}
  onPause={() => console.log("Video is paused")}
  onEnded={() => console.log("Video has ended")}
  onTimeUpdate={(time) => console.log("Time is updated", time)}
/>

Note: If you want to build a player with your own Completely Custom UI. You can check out the Building a custom player guide in the reactjs-media documentation.

Conclusion

In this article, we have learned how to create a responsive custom video player in React using the reactjs-media library. This library provides a set of components and a simple API to create a customizable media player for your application. You can explore more features and customization options by checking out the reactjs-media documentation.

Creating a URL Shortener service in Python Django

Beingana Jim Junior — Mon, 08 Aug 2022 13:57:03 +0000

URL shorteners have become a popular service on the web. Companies like bitly are making great fortunes from them. But sometimes when you want a custom URL you get to pay for the service. So in this tutorial I am going to show you how to build a URL shortener service in Django.

In this tutorial I expect you to be familiar with templates and forms because i wont cover them but instead i will just show you how to pragmatically build create the service in python code.

Setup

To setup the project, we shall need only one package. django-urlshortner. So lets begin by installing the package using pip. This asumes you have already setup a django project.

pip install django-urlshortner

After you should add the urlshortner app to your INSTALLED_APPS in settings.py

Configuration

INSTALLED_APPS = [
    # ....
    "urlshortner"
]

Then you migrate the models to your database

python3 manage.py migrate

Lastly add the routes to your URLConf in your urls.py of your project.

url_patterns = [
    # ...
    path("r/", include("urlshortner.urls")),
]

Now you are good to go.

Usage

The library provides a list of utils to create shortened urls.

To create a short version of a url use the shorten_url function from urlshortner.utils module

# python3 manage.py shell
from urlshortner.utils import shorten_url

url_route = shorten_url(
    "https://github.com/jim-junior/django-urlshortner",
    is_permanent=False
)

print(url_route)
# >>> 0ee3f0

You can now navigate to you the route that you assigned to urlshortner.urls in your URLConf add the returned value att the end of the url. In this case it would be http://localhost:8000/r/0ee3f0/ and this would redirect you to the right URL

Sometimes you want to create a custom URL. For example you want to create a short link for a blog about your new product and you want a url that is easy to remember. You can add this easily by adding the value argument to the shorten_url function

from urlshortner.utils import shorten_url

url_route = shorten_url(
    "https://myblog.com/blog/2022/10/10/..../my-new-product",
    value="NewProduct"
    is_permanent=False
)

You can now navigate to https://localhost/r/NewProduct and It will redirect you

Now that you know how to use the package i think you can intergrate it with you project.

Hope this article was helpful. You can git the project a star on Github or if you have any idea to add on you can contribute to its repository. And bythaway I am the Author of this library.

Forem: Beingana Jim Junior

Implementing Resource Versioning in Conveyor CI

How it is designed in other systems

Intended Design in Conveyor CI

Technical Implementation

Compact Strategy

A Complete Guide to etcd: The Distributed Key-Value Store Powering Cloud Infrastructure

Installation

Installing on a host machine

For Linux & MacOS

For Windows

Installing as a Container

Setting up an etcd cluster

Working with etcd

Interacting with etcd

Querying etcd

Storing data to etcd

Reading data

Deleting data

How data is stored in etcd

Operating etcd

Cluster health

Snapshots

Monitoring etcd

Metrics

Logging

Conclusion

Designing The Conveyor CI Pipeline Engine

Implementing Pipelines

Use case Workflows

How the Engine Works

Execution Workflow

A Simple Diagram representing this process

Technical Details

Conclusion

A Guide to contributing to the Conveyor CI Driver Runtime

Introduction

How the Driver Runtime Works

What is the Driver Runtime

What Features does it Provide

Driver runtime Components

Driver Manager

Driver Logger

Driver

Client Library

Component Diagram

Contributing Guide

Environment Setup

Workflow

Reporting Bugs

Suggesting Enhancements

Submitting Code Changes

Improving Tests or Documentation

Resources

Designing the Conveyor CI Scheduling Mechanism

How the Mechanism works

Preventing resource contention

Horizontal Scaling

Conclusion

How to deploy an anonymous website on the Dark Web

The Onion Router (Tor)

How it works

Deploying on the Tor Network

Installing Tor Services

Setting up your Web Application/Server

Add the Website to Tor

Running the Tor Service

An Introduction to Supervised Learning Models

What is Supervised Learning

Problems in Supervised learning

Supervised Learning Models

Model Inputs

Model Outputs

Conclusion

A Practical Introduction to the Event Driven Architecture

How the Event-Driven Architecture Works

Concepts

The Event broker

Event Portals

Event Mesh

Interacting with `etcd`