Forem: Richard Kovacs

The Kubernetes Performance Challenge, Join to the contest!

Richard Kovacs — Wed, 04 Mar 2026 12:27:51 +0000

We were interested in how far we can go with Kubernetes. While it can scale up applications to the space (beyond the clouds 😜), Kubernetes itself fails to scale gracefully. Or maybe not?

Think your cluster is fast? Beat or cluster!

The contest is running on GitHub.

🚀 The Challenge

Participants are invited to execute standardized performance tests against the Kubernetes to measure latency, throughput, and system stability.

Participation is free and requires no purchase.

Check out Current Leaderboard

🏆 The Grand Prize

The ultimate reward for pushing the boundaries of cloud engineering.

Every participant who achieves a higher score than ours will receive a custom-manufactured, limited-edition technical shirt.

The Design:

Front: Kubernetes Performance Challenge
Back: I BEAT HARIKUBE
Sleeve: Your specific performance metrics (like number of objects, latencies, and success rate) printed as a permanent record of your achievement.

🎖️ The "First Ten" Grant

The first 10 participants to submit a valid benchmark report (regardless of their final ranking) will receive a exclusive one-year HariKube Pro Edition License.

🎲 The "Second Chance" Draw

Don't be discouraged if you missed the first ten or couldn't beat our score. At the conclusion of the challenge, one additional HariKube Pro Edition License will be raffled off among all participants who submitted a valid report but haven't won through the other categories.

📅 Contest Period

Starts at 2026.02.25. 18:00 GMT
Ends at 2026.05.25. 22:00 GMT

📜 Rules and Constraints

To ensure valid and comparable results, all participants must adhere to the following constraints:

Discussion Protocol: Upon completion of the challenge, all winners (who beat our score) will be invited to a friendly discussion regarding the results, architectural insights, and the underlying optimization strategies. This ensures a high-level technical exchange within a vetted peer group.
RBAC Policy Enforcement: Role-Based Access Control must remain enabled and fully functional. Performance gains achieved by bypassing security layers are disqualified.
Feature Parity: All core Kubernetes features, including AMission Controllers and standard API validation, must be active. No "stripping" of the environment is permitted.
No Shortcuts: Any modification to the core that compromises data consistency will result in an invalid run.
Zero Tolerance for Instability: Database or Kubernetes crash lead to immediate disqualification.

Integrity & Honor: We are aware that benchmarks can be manipulated, and no amount of technical restriction can fully prevent dedicated cheating. However, this challenge is a matter of professional honor. We are not fools - any discrepancies will immediately surface during the discussion. Let’s not ruin the collective experience for 15 minutes of fame; if you can’t beat the baseline honestly, use the opportunity to learn how to improve.

Winners are responsible for any local taxes or customs duties related to their prizes.
HariKube reserves the right to disqualify any submission that violates the spirit of the challenge. Our decision is final and non-negotiable.
By submitting a PR, you consent to the public display of your GitHub username on the leaderboard. Personal information for prizes will be handled temporary and will not be shared with third parties.
Legal Disclaimer: This is a skill-based challenge. Participation is free. By submitting your results, you acknowledge that the final validation of any score is subject to the Technical Discussion Protocol to ensure compliance with the rules.

Score calculation

floor(checks_total / (http_req_duration.med*0.2 + http_req_duration.p90*0.3 + http_req_duration.p95*0.5) * pow(1 - checks_failed / checks_total, 7))

http_req_duration counted in ms

🛠️ Getting Started

Deployment: Bring your own Kubernetes.
Benchmarking: Execute our k6 test against on your cluster.
- Run make command for details.
Create a PR: Share your results via Pull-Request on the challenge repository. The PR must contains:
- The data is collected under results/${USER} directory
- Summary of k6 benchmark execution, example location: results/${USER}/1771941557_20_sum.json
- Collected benchmark data, example location: results/${USER}/1771941557_20.json
- Updated Current Leaderboard in README.md
- If you have the highest score; update Leader Benchmark Details in README.md based on the example

🕵️ Benchmark Details

The test execution is 1 hour
Define concurrency on your own taste
Each iteration creates 6 different type of custom resources in parallel
Each iteration reads back the created custom resources via label selector in parallel

🔀 Join to the Contest

For more details please visit the GitHub page where contest is running.

Beyond the Cluster: Rethinking Your Kubernetes Platform Strategy

Richard Kovacs — Tue, 27 Jan 2026 11:23:16 +0000

💡 Kubernetes is a "platform for building platforms," not an end-state solution

As we navigate the technological landscape of 2026, the industry has reached a pivotal realization: Kubernetes was never meant to be the final destination for developers. While it has become the undisputed standard for orchestration, treating a raw Kubernetes cluster as an end-state solution often leads to "configuration fatigue" and fragmented architectures. The top trend this year is the aggressive shift toward Application Modernization through specialized Platform-as-a-Service (PaaS) layers built on top of the Kubernetes control plane.

The modern goal is to achieve a true separation of concerns. Organizations no longer want their developers bogged down by the complexities of container networking, storage classes, or boilerplate YAML. Instead, 2026 is the year of the CNPaaS (Cloud-Native-Platform-as-a-Service) - a paradigm where the underlying infrastructure is abstracted into a seamless, intelligent operating system. This evolution transforms Kubernetes from a complex toolbox into a unified, invisible engine that powers business logic, allowing teams to deliver pure value at a pace that matches the speed of the digital economy.

👷 The "Day 2" Reality Check

If you’ve successfully provisioned a cluster, configured your first CNI, and seen that glorious green "Running" status across your pods. You’ve conquered Day 1. But as many platform teams are discovering in 2026, Day 1 is the honeymoon phase. The real struggle begins with Day 2 operations, where the focus shifts from "How do we run this?" to "How do our developers actually use this without burning out?"

Rethinking your strategy for Day 2 means acknowledging that Kubernetes must evolve into a developer-centric service. It’s about moving beyond raw orchestration and asking:

How do we make the infrastructure "invisible"?
How do we unify disparate service designs (functions, microservices, and APIs) into a single, cohesive experience?
How do we ensure the control plane remains a source of truth rather than a bottleneck?
How do we deliver the platform to developer boxes?

In this phase, the goal isn't just to keep the lights on, it's to transform your cluster into a high-performance PaaS where the platform does the heavy lifting, letting your developers focus on the only thing that truly matters: delivering business value.

🛤️ The "Options" Landscape: Finding the Right Path for 2026

without the need for completeness

When teams hit the Day 2 wall, they typically look for one of three strategic escape hatches. Each offers a different trade-off between control and complexity.

1. 🏗️ The "Build Your Own" Path

Many organizations start by trying to "hand-stitch" a platform using a mix of open-source scripts, massive Helm charts, and custom Glue-code.

The Pros: Complete control. You can build exactly what your unique business logic requires without any external opinionated constraints.
The Cons: It is an operational nightmare. You aren't just building an app; you are now maintaining a custom "distribution" of Kubernetes, with the need of supporting Hybrid Cloud. This requires a dedicated team of 5–10 engineers just to keep up with quarterly releases, security patches, and the "technical debt" of your own abstractions. Typically, it is challenging to run on developer's machine and developers need extra effort to learn how it works.

2. 🎩 The "Application Runtime" Path

A good example of this approach is Dapr (Distributed Application Runtime), it attempts to solve the developer experience by providing "building block" APIs via sidecars.

The Pros: Excellent developer abstraction. It makes code portable and provides standardized patterns for state, pub/sub, and secrets across any language.
The Cons: Infrastructure Overhead. Dapr requires its own separate stack of infrastructure. Because it uses the sidecar pattern, every single pod in your cluster carries the "sidecar tax" - consuming extra CPU and memory. Furthermore, to actually function, it relies on external dependencies like Redis or Kafka that you still have to manage, scale, and secure independently of Kubernetes. It should be a vendor lock in and developers need to learn in details.

3. 🏋️‍♂️ The "Virtual Control Plane" Path

Solutions like kcp or vCluster are CNCF projects that aims to scale/isolate the Kubernetes API by creating logical/virtual "workspaces," effectively providing Control-Plane-as-a-Service.

The Pros: Incredible multi-tenancy. It allows you to provide thousands of isolated API endpoints to different teams without spinning up thousands of physical clusters. They are transparent and achieve true separation of concerns - giving developers a transparent, framework-free experience at the intentional cost of increasing the operator's headache to manage a centralized, high-level control plane.
The Cons: The Filtering Bottleneck. While kcp and vCluster scale the quantity of APIs, they still struggle with the depth of data. Crucially, they lack native storage-side filtering. This means that when a developer or a controller queries a large set of resources, the platform often has to handle heavy data filtering on the client-side. This results in high memory pressure and latency spikes when your "Day 2" state begins to grow into the thousands of objects. They can be transparent on developer's machines, but in production they are increasing infrastructure complexity.

The Open Source Bridge: To address this storage-side filtering bottleneck, HariKube has released an open-source edition that ships with a specialized version of vCluster integrated with both storage-side filtering and garbage-collection. By replacing the default data store with a SQL-backed engine (via a patched Kine/Kubernetes build), this version allows virtual clusters to offload query filtering to the database level. Build Your Kubernetes Platform-as-a-Service Today

4. 🛸 The "Smart Middleware" Path

If custom builds are too heavy, sidecars are too invasive, and multi Kubernetes is expensive, HariKube offers a surgical alternative. Instead of building on top of Kubernetes, this approach injects smart routing directly into the control plane's data flow. HariKube acts as a high-performance transparent proxy between the Kubernetes API and your data. By replacing the rigid etcd backend with a Dynamic Data Layer, it transforms the Kubernetes API into a database-agnostic Single Source of Truth.

The Pros: It provides a radical "Double-Win." You get the developer-friendly features of Dapr (like event-driven triggers, service discovery and unified service patterns) and the multi-tenancy of kcp/vcluster. And the good news, they all built in Kubernetes, so you don't need introduce an abstraction on top of Kubernetes. Your applications become Cloud-Native by Design. It supports storage-side filtering and garbage-collection, meaning your platform remains lightning-fast even as your Day 2 state grows to hyperscale.
The Cons: The real change here isn't about operational burden; it’s about perspective. You stop viewing Kubernetes as an infrastructure silo and start treating it as a high-performance document warehouse. The shift is learning to stop reaching for external tools and instead using Kubernetes built-ins - CRDs, the Aggregation API, and Controller loops - to serve application logic directly. Adopting this model does add specific operational considerations: for instance, it relies heavily on Webhooks. While webhooks introduce extra hops, they are also a hidden "Pro" - they force your application to be horizontally Scalable by design.

🏁 Final Verdict: Which Path is Yours?

As we navigate the platform trends of 2026, the question is how you want to manage your complexity:

Choose Dapr if you prefer application-level abstractions and are comfortable managing a secondary infrastructure stack and the associated "sidecar tax."
Choose kcp/vCluster if your priority is logical partitioning and your data sets are small enough to handle the lack of storage-side filtering.
Choose open source HariKube if you can partition your data by control plane and rely on storage-side filtering to manage massive datasets with minimal overhead.
Choose full-powered HariKube if you're prepared to embrace Kubernetes as your native data and logic engine, eliminating manual overhead by letting the platform itself drive your high-scale operations.

Strategy	Core Approach	Best For	Key Pros	Key Cons
Build Your Own	hand-crafted scripts, Helm, custom glue-code	100% custom control	no opinionated constraints	fully custom; requires 5–10 dedicated engineers
Application Runtime	sidecar-based APIs	standard patterns across multiple languages	portable code	sidecar tax; requires managing external dependencies
Virtual Control Plane	isolated APIs	multi-tenancy & isolation	transparent for developers	lack of storage-side filtering
Smart Middleware	transparent data proxy	standard patterns across multiple languages, isolation	portable code, transparent for developers	commercial product for businesses

That’s it! If you’re ready to embrace the mental shift and explore the hyperscale, unified service model that HariKube offers, follow the link here to The Future of Kubernetes PaaS and Kubernetes-native Service Development is Here. By turning Kubernetes into your primary logic and data engine, you can finally stop managing infrastructure and start delivering pure business value.

Thank you for reading, and feel free to share your thoughts.

Build Your Kubernetes Platform-as-a-Service Today

Richard Kovacs — Thu, 06 Nov 2025 13:15:19 +0000

💡 Our Mission at HariKube: Unlocking Kubernetes' True Potential

The promise of Kubernetes was simple: to become the universal operating system for the cloud. Yet, for all its power, it hit a wall at enterprise scale—specifically, the limitations of ETCD and a fragmented application development experience. The result? A fantastic container orchestrator, but a broken Platform-as-a-Service (PaaS).

We believe that the Kubernetes API should be the single, scalable, and language-agnostic foundation for every cloud-native workload—from event-driven nanoservices to complex stateful microservices and classic REST APIs. Read the full article [→]

🛑 The Scalability Wall: ETCD Limits and Client-Side Filtering

While Kubernetes is masterful at scheduling and orchestration, its control plane has two fundamental design decisions that prevent it from scaling linearly into a true enterprise PaaS: the reliance on ETCD and the practice of client-side filtering. ETCD is officially recommended to stay near 2GB of total data. For small resources, this is adequate, but for large-scale enterprise clusters managing tens of thousands of custom resources (CRs), the limit is quickly hit.

The second, and often more insidious, scalability issue stems from how the Kubernetes API server queries data:

🐌 Kubernetes API Server fetches all relevant resources from the database (ETCD), then filters them in memory (on the client side) before returning the final result.

🚀 HariKube's Dynamic Data Fabric: Scaling to Infinity

HariKube introduces a dynamic, database-agnostic data fabric that completely reimagines how the Kubernetes API server interacts with its persistent store. This fabric unlocks Kubernetes’ true potential, providing the enterprise-level scaling needed to manage massive clusters and high-throughput microservices.

However, we understand that not all use cases demand this full enterprise architecture. To empower the wider community and allow everyone to benefit from these advancements, we have Ppen-Sourced a critical toolset designed to help developers and smaller organizations achieve robust scalability and avoid common ETCD limitations by implementing efficient storage-side filtering within their existing Kubernetes environments.

🛠️ Introducing the HariKube Open Source Toolset

To democratize the advancements needed to overcome the limitations of ETCD and client-side filtering, we have open-sourced a core toolset. This solution acts as a bridge, allowing standard Kubernetes deployments to use a scalable SQL backend and benefit from storage-side filtering without adopting the full enterprise architecture.

This toolset consists of two key components:

Modified Kine (Kine is not etcd) Kine is an existing etcd-shim (a compatibility layer) that translates the Kubernetes API server's native ETCD API calls into standard SQL queries for backends like SQLite, Postgres, or MySQL. It allows Kubernetes to use these databases instead of ETCD.

Our Modification: The HariKube version of Kine contains crucial enhancements that implement storage-side filtering logic. When the Kubernetes API server issues a request that typically requires client-side filtering (like a label selector), our modified Kine translates that request into an optimized SQL WHERE clause. This offloads the filtering work to the efficient SQL database, drastically cutting down on the amount of data transferred and processed by the API server, directly solving the client-side filtering performance bottleneck.

Modified Kubernetes Control Plane While Kine handles the data access layer, to fully leverage the performance gains, we also provide a small set of targeted modifications to the Kubernetes API Server itself.

The Change: These modifications enhance the way the API server constructs requests to its storage layer (our modified Kine). Specifically, they ensure that the server-side filters (like the label selectors and field selectors) are correctly passed down and translated by Kine into the SQL query, rather than being handled as an afterthought in memory.

The Result: This coupling of the modified Kine and the modified Kubernetes Control Plane provides a transparent, drop-in replacement for the standard ETCD setup, enabling a huge leap in scalability and query performance for clusters that choose to adopt a SQL backend.

vCluster Integration (Isolation and Scalability) While it is entirely possible to run a standalone instance of this setup, for real-world production use, we strongly suggest utilizing the vCluster version of the toolset.

What is vCluster? vCluster (Virtual Cluster) is an open-source tool that provisions lightweight, isolated Kubernetes control planes (the virtual cluster) running inside a regular Kubernetes namespace (the host cluster).

The Benefit: Deploying our modified control plane and Kine setup inside a vCluster provides critical advantages:

Separation of Concerns: It cleanly separates the underlying infrastructure data (the host cluster) from the application and service data (the virtual cluster), enhancing stability and security.
Easier Deployment and Management: Virtual clusters are simple to provision and manage declaratively, just like any other Kubernetes resource.
Individual Scalability: Each virtual cluster gets its own Kubernetes API server and isolated Kine/SQL backend, meaning its scalability and performance are independent of all other clusters and the host clusters, making it the ideal pattern for deploying multiple scalable services.

By using this open-source toolset, developers can immediately start building more data-intensive microservices and operators without worrying about the looming scalability wall of a standard ETCD deployment.

⚙️ Step 1: Bring Your Cluster

The power of the HariKube open-source toolset lies in its agnosticism—it is designed to work with any existing Kubernetes distribution you use today. There is no vendor lock-in and no need to migrate to a specific cloud provider or distribution.

Whether you are running:

Cloud-Managed Clusters: Such as GKE, EKS, AKS, or DigitalOcean.
On-Premises or Self-Hosted Clusters: Using k3s, RKE, or bare metal deployments.
Local Development Environments: Like minikube or kind.

...the principle remains the same. You bring your existing cluster, and our modified control plane and Kine shim will integrate directly with it. This allows you to immediately begin scaling your control plane and benefiting from storage-side filtering without any disruptive infrastructure changes.

✨ Step 2: Launch the Scalable Control Plane

With your host Kubernetes cluster ready, the next step is to deploy the vCluster instance containing our modified Kine shim and Kubernetes control plane. This single declarative step handles the creation of the virtual cluster and configures it to use a scalable, file-backed SQLite instance (perfect for testing and small deployments) instead of the standard ETCD.

This deployment instantly gives you a new, isolated control plane that benefits from:

ETCD Avoidance: State is routed to the SQL backend via Kine.
Storage-Side Filtering: All data queries are optimized for performance.
Isolation: The control plane is separated into its own namespace via vCluster.

To execute this, simply run the following command:

kubectl apply -f https://github.com/HariKube/kine/releases/download/release-v0.14.6/vcluster-kine-sqlite.yaml

Once the virtual cluster is running, you can connect to it directly using the vCluster CLI:

vcluster connect kine

You are now connected to a highly performant, isolated control plane that is no longer limited by ETCD or client-side filtering. Congratulations on taking the first step towards truly scalable cloud-native development!

Once connected, vCluster changes your current local KUBECONFIG file to point directly to the virtual control plane. This is seamless for developers and means all subsequent kubectl commands will interact with your new, scalable instance. When you are finished working on the virtual cluster, you can easily swap back to your host cluster by running the command vcluster disconnect.

🧠 Final Thoughts

The journey to an unconstrained, scalable Kubernetes platform begins here. The limitations imposed by ETCD and the inefficiency of client-side filtering have long bottlenecked enterprise adoption and prevented Kubernetes from fully realizing its potential as a true PaaS.

This open-source release is a testament to our mission: to redefine the cloud-native application development blueprint. Whether you adopt the full HariKube enterprise data fabric or utilize these open-source tools, you are moving toward a more resilient, high-throughput, and scalable future. Start building your next generation of microservices, operators, and functions where the Kubernetes API truly is the scalable source of truth, no longer limited by yesterday's architectural constraints.

🙏 Share Feedback and Report Issues

Your feedback is invaluable in helping us improve this operator. If you encounter any issues, have a suggestion for a new feature, or simply want to share your experience, we want to hear from you!

Report Bugs: If you find a bug, please open a GitHub Issue. Include as much detail as possible, such as steps to reproduce the bug, expected behavior, and your environment (e.g., Kubernetes version).
Request a Feature: If you have an idea for a new feature, open a GitHub Issue and use the enhancement label. Describe the use case and how the new feature would benefit the community.
Ask a Question: For general questions or discussions, please use the GitHub Discussions.

That’s it! You've successfully deployed a fully isolated, performance-optimized, and scalable virtual Kubernetes control plane. It's now ready to host your most demanding applications, operators, and event-driven functions, free from the constraints of ETCD limits and client-side filtering.

Thank you for reading, and feel free to share your thoughts.

Turn any of your APIs into a Kubernetes-native citizen

Richard Kovacs — Thu, 06 Nov 2025 12:00:16 +0000

⚡ Transforming APIs into Kubernetes-Native Citizens

The power of Kubernetes lies in its unified control plane and its resource model. But what if you could make any of your existing APIs or services behave like a native Kubernetes resource—complete with kubectl compatibility, cluster-level governance, and seamless integration?

Developing extensions via the Kubernetes Aggregation Layer has traditionally been a highly complex task, demanding deep knowledge of internal API structures, serialization, and resource handling. This steep learning curve often prevents developers from extending Kubernetes as much as they'd like.

This post introduces the kubernetes-aggregator-framework, an open-source tool designed to drastically simplify this process. We'll show you how this framework hides the complexity of the Kubernetes API server internals, allowing you to rapidly turn existing microservices or custom data sources into Kubernetes-native citizens.

❓ What the Kubernetes Aggregation Layer Gives You

The Kubernetes Aggregation Layer allows you to extend the core Kubernetes API with your own custom APIs. Think of it as a gateway that lets you add new types of resources to your cluster that behave just like native Kubernetes objects (like pods or services).

🤹 The Difficulty of Developing an API Server Without Frameworks

Building a custom API server for Kubernetes from scratch is a complex and challenging task. It requires a deep understanding of the Kubernetes API's internals, including:

This level of detail makes it very difficult to develop an API server without extensive, in-depth knowledge of Kubernetes' inner workings.

💡 How This Framework Simplifies the Process

Recognizing these challenges, our framework provides a straightforward solution to make building custom Kubernetes API servers easy and accessible. It hides away much of the complex, low-level work, allowing you to focus on your core logic.

🔌 How to Use It

Kubernetes uses HTTPS by default, so you need a certificate for the service. The most common way to automate this step is installing Cert-Manager to the cluster.

In the first step create an APIService and the related service manifests. Edit your Deployment to mount certificate.

apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  name: v1.custom-api.example.com # This has a stict format
  annotations:
    cert-manager.io/inject-ca-from: v1-custom-api-service/defaultyour service
spec:
  group: custom-api.example.com
  version: v1
  groupPriorityMinimum: 1000
  versionPriority: 10
  service:
    name: v1-custom-api-service
    namespace: default
    port: 443
  caBundle: ""
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: v1-custom-api-service-cert
  namespace: system
spec:
  secretName: v1-custom-api-service-tls
  issuerRef:
    name: v1-custom-api-service-ca
  commonName: v1-custom-api-service/default.svc
  dnsNames:
  - v1-custom-api-service/default.svc
  - v1-custom-api-service/default.svc.cluster.local
---
apiVersion: v1
kind: Service
metadata:
  name: v1-custom-api-service
spec:
  ports:
    - port: 443
      protocol: TCP
      targetPort: 7443
  selector:
    app: custom-api.example.com

Creating raw endpoints, without any Kubernetes behaviour dependency

server := *kaf.NewServer(kaf.ServerConfig{
    Port:     port,
    CertFile: certFile,
    KeyFile:  keyFile,
    Group:    "custom-api.example.com",
    Version:  "v1",
    APIKinds: []kaf.APIKind{
        {
            ApiResource: metav1.APIResource{
                Name:  "foo",
                Verbs: []string{"get"},
            },
            RawEndpoints: map[string]http.HandlerFunc{
                "": func(w http.ResponseWriter, r *http.Request) {
                    ...
                },
                "/bar": func(w http.ResponseWriter, r *http.Request) {
                    ...
                },
            },
        },
    },
}),

_ := server.Start(context.Background())

Call API via kubectl.

kubectl get --raw /apis/custom-api.example.com/v1/foo
kubectl get --raw /apis/custom-api.example.com/v1/foo/bar

Create fully customized API endpoints for cluster and namespace scoped custom resource

kaf.APIKind {
    ApiResource: metav1.APIResource{
        Name:  "clustertasks",
        Kind:  "ClusterPod",
        Verbs: []string{"get", "list", "watch", "create", "update", "delete"},
    },
    CustomResource: &kaf.CustomResource{
        CreateHandler: func(namespace, name string, w http.ResponseWriter, r *http.Request) {
            w.Header().Set("Content-Type", "application/json; charset=utf-8")
        },
        GetHandler: func(namespace, name string, w http.ResponseWriter, r *http.Request) {
            w.Header().Set("Content-Type", "application/json; charset=utf-8")
        },
        ListHandler: func(namespace, name string, w http.ResponseWriter, r *http.Request) {
            w.Header().Set("Content-Type", "application/json; charset=utf-8")
        },
        ReplaceHandler: func(namespace, name string, w http.ResponseWriter, r *http.Request) {
            w.Header().Set("Content-Type", "application/json; charset=utf-8")
        },
        DeleteHandler: func(namespace, name string, w http.ResponseWriter, r *http.Request) {
            w.Header().Set("Content-Type", "application/json; charset=utf-8")
        },
        WatchHandler: func(namespace, name string, w http.ResponseWriter, r *http.Request) {
            w.Header().Set("Content-Type", "application/json; charset=utf-8")
        },
    },
},
kaf.APIKind {
    ApiResource: metav1.APIResource{
        Name:       "customtasks",
        Namespaced: true,
        Kind:       "CustomPod",
        Verbs:      []string{"get", "list", "watch", "create", "update", "delete"},
    },
    CustomResource: &kaf.CustomResource{
        CreateHandler: func(namespace, name string, w http.ResponseWriter, r *http.Request) {
            w.Header().Set("Content-Type", "application/json; charset=utf-8")
        },
        GetHandler: func(namespace, name string, w http.ResponseWriter, r *http.Request) {
            w.Header().Set("Content-Type", "application/json; charset=utf-8")
        },
        ListHandler: func(namespace, name string, w http.ResponseWriter, r *http.Request) {
            w.Header().Set("Content-Type", "application/json; charset=utf-8")
        },
        ReplaceHandler: func(namespace, name string, w http.ResponseWriter, r *http.Request) {
            w.Header().Set("Content-Type", "application/json; charset=utf-8")
        },
        DeleteHandler: func(namespace, name string, w http.ResponseWriter, r *http.Request) {
            w.Header().Set("Content-Type", "application/json; charset=utf-8")
        },
        WatchHandler: func(namespace, name string, w http.ResponseWriter, r *http.Request) {
            w.Header().Set("Content-Type", "application/json; charset=utf-8")
        },
    },
},

Call API via kubectl.

kubectl create --raw /apis/custom-api.example.com/v1/clustertasks/foo -f clustertask.yaml
kubectl replace --raw /apis/custom-api.example.com/v1/clustertasks/foo -f clustertask.yaml
kubectl get clustertasks
kubectl get clustertasks foo
kubectl get clustertasks -w
kubectl get clustertasks foo -w
kubectl delete clustertasks foo

kubectl create --raw /apis/custom-api.example.com/v1/namespaces/default/customtasks/foo -f customtask.yaml
kubectl replace --raw /apis/custom-api.example.com/v1/namespaces/default/customtasks/foo -f clustertask.yaml
kubectl get customtasks
kubectl get customtasks foo
kubectl get customtasks -w
kubectl get customtasks foo -w
kubectl delete customtasks foo

Create and API extending Kubernetes API capabilities, for example collecting events of Pods

type CombinedPod struct {
    metav1.TypeMeta   `json:",inline"`
    metav1.ObjectMeta `json:"metadata,omitempty"`

    Spec   corev1.PodSpec    `json:"spec,omitempty"`
    Status corev1.PodStatus  `json:"status,omitempty"`
}

type CombinedPodList struct {
    metav1.TypeMeta `json:",inline"`
    metav1.ListMeta `json:"metadata,omitempty"`
    Items           []CombinedPod `json:"items"`
}

kubeClient, _ := client.New(ctrl.GetConfigOrDie(), client.Options{
  Scheme: scheme,
})

dynamicKubeClient, _ := dynamic.NewForConfig(ctrl.GetConfigOrDie())

Server: *kaf.NewServer(kaf.ServerConfig{
    KubeClient: kubeClient,
    DynamicKubeClient: dynamicKubeClient,
    ...

kaf.APIKind {
    ApiResource: metav1.APIResource{
        Name:       "combinedpods",
        Namespaced: true,
        Kind:       "CombinedPod",
        Verbs:      []string{"get", "list", "watch"},
    },
    Resource: &kaf.Resource{
        CreateNew: func() (schema.GroupVersionResource, client.Object) {
            return corev1.GroupVersion.WithResource("pods"), &corev1.Pod{}
        },
        CreateNewList: func() (schema.GroupVersionResource, client.ObjectList) {
            return corev1.GroupVersion.WithResource("podlist"), &corev1.PodList{}
        },
        ListCallback: func(ctx context.Context, namespace, _ string, objList client.ObjectList) (any, error) {
            podList, ok := objList.(*corev1.PodList)
            if !ok {
              return nil, fmt.Errorf("failed to convert podlist for: %s", objList.GetObjectKind().GroupVersionKind().String())
            }

            // Do what you want

            combinedPods := CombinedPodList{
              TypeMeta: metav1.TypeMeta{
                Kind:       "CombinedPodList",
                APIVersion: Group + "/" + Version,
              },
              ListMeta: metav1.ListMeta{
                ResourceVersion:    podList.ResourceVersion,
                Continue:           podList.Continue,
                RemainingItemCount: podList.RemainingItemCount,
              },
              Items: []CombinedPod{},
            }

            for _, t := range items {
              pod := t.(*corev1.Pod)

              ct := CombinedPod{
                TypeMeta: metav1.TypeMeta{
                  Kind:       "CombinedPod",
                  APIVersion: Group + "/" + Version,
                },
                ObjectMeta: pod.ObjectMeta,
                Spec:       pod.Spec,
              }

              combinedPods.Items = append(combinedPods.Items, ct)
            }

            return combinedPods, nil
          },
          WatchCallback: func(ctx context.Context, _, _ string, unstructuredObj *unstructured.Unstructured) (any, error) {
            pod := corev1.Pod{}
            if err := runtime.DefaultUnstructuredConverter.FromUnstructured(unstructuredObj.Object, &pod); err != nil {
              return nil, fmt.Errorf("failed to convert unstructured: %v", err)
            }

            // Do what you want

            cp := CombinedPod{
              TypeMeta: metav1.TypeMeta{
                Kind:       "CombinedPod",
                APIVersion: Group + "/" + Version,
              },
              ObjectMeta: pod.ObjectMeta,
              Spec:       pod.Spec,
            }

            return cp, nil
          },
      },
},
{{< /code >}}

Call API via `kubectl`.

{{< code bash >}}kubectl get combinedtasks -A
kubectl get combinedtasks -n default
kubectl get combinedtasks -n default foo
kubectl get combinedtasks -n default foo -o yaml

🧠 Final Thoughts

We've explored how the kubernetes-aggregator-framework dramatically simplifies the complex world of the Kubernetes Aggregation Layer. The ability to expose any of your existing APIs as native Kubernetes Custom Resources is a game-changer.

By abstracting away the heavy lifting—like handling API registration, request routing, serialization, and proper Kubernetes API conventions—the framework allows you to focus purely on your core business logic. You gain the best of both worlds: the simplicity of standard API development coupled with the powerful, unified control plane that Kubernetes offers.

This framework is not just about making development easier; it’s about enhancing your cluster’s capabilities. You can now enforce consistent security policies, leverage standard tooling like kubectl, and achieve genuine Kubernetes-native integration for every microservice you build.

We hope this tool empowers you to unlock new levels of performance and simplicity within your data topology. Don't let the complexity of Kubernetes API extensions hold you back any longer.

🙏 Share Feedback and Report Issues

Your feedback is invaluable in helping us improve this framework. If you encounter any issues, have a suggestion for a new feature, or simply want to share your experience, we want to hear from you!

Report Bugs: If you find a bug, please open a GitHub Issue. Include as much detail as possible, such as steps to reproduce the bug, expected behavior, and your environment (e.g., Kubernetes version, Go version).
Request a Feature: If you have an idea for a new feature, open a GitHub Issue and use the feature request label. Describe the use case and how the new feature would benefit the community.
Ask a Question: For general questions or discussions, please use the GitHub Discussions.

This project is part of our journey to transform Kubernetes into a true Platform-as-a-Service. We not just dream about this but built for you. Read details here: [→].

That’s it! You've seen how the kubernetes-aggregator-framework completely removes the friction from extending your Kubernetes control plane. It's time to stop treating your APIs as external services and start integrating them as first-class Kubernetes resources.

Thank you for reading, and feel free to share your thoughts.

The Future of Kubernetes PaaS and Kubernetes-native Service Development is Here

Richard Kovacs — Mon, 27 Oct 2025 18:01:09 +0000

We at HariKube strongly believe in the transformative power of cutting-edge technology to shape the future of service development. As the digital landscape continues to evolve at an unprecedented pace, the need for robust, scalable, and efficient Cloud-Native infrastructure becomes paramount. We are committed to turning Kubernetes into a real Platform-as-a-Service (PaaS), where it not just orchestrates individual services, but becomes the source of truth for the entire platform. Join us as we delve into the exciting advancements and upcoming trends that define ...

The Future of Kubernetes and Service Development

This vision leads us to a fundamental shift: Imagine Kubernetes itself evolving beyond an orchestrator and becoming the definitive platform. We're talking about a single control plane for managing not just containers, but also serving serverless functions, operator based microservices, and all your custom APIs. This is achievable now. By combining HariKube’s innovative dynamic data layer, automated serverless function triggers, intelligent operator loops, and a smart utilization of the Kubernetes Aggregation API, we are ushering in a world where Kubernetes is the singular, intelligent operating system for your entire Cloud-Native estate.

The dynamic data layer is a critical component designed to address the inherent scalability limitations of Kubernetes' native data store, ETCD. Instead of relying solely on the single, clustered ETCD instance for all cluster state, this dynamic data layer effectively replaces ETCD by allowing you to define a user-defined database agnostic topology (flat or hierarchical), chosing any number of the supported databases (9 total including MySQL, PgSQL, etc.).

This comprehensive approach achieves a powerful double-win: we are simultaneously unifying disparate service designs across your system and eliminating Kubernetes' main scalability bottleneck. This is the revonutionaly architectural shift: by liberating the control plane from the constraints of single-instance state management, HariKube unlocks true hyperscale for every Kubernetes deployment. By tackling these two critical areas, Kubernetes finally evolves into a true Cloud-Native-Platform-as-a-Service (CNPaaS). This transformation is key to driving essential separation of concerns: developer teams can now focus entirely on delivering core business logic, while infrastructure teams can dedicate their expertise to managing and optimizing the underlying infrastructure and platform capabilities. Infrastructure complexity becomes an API, not a configuration hurdle. This singular focus accelerates innovation and cuts time-to-market dramatically. Developers are no longer bogged down writing boilerplate YAML, taking care on self-managed databases, managing storage classes, or debugging platform issues—they simply consume the functionalities built-in Kubernetes.

The result is a streamlined, resilient platform that offers not just scale, but agility. By empowering development teams with clean interfaces and giving platform teams powerful, flexible tools, HariKube achieves the ultimate goal: making the Kubernetes platform disappear, leaving only pure business value behind.

The Single Source of Truth: Unified Service Design

With HariKube, Kubernetes is no longer just a platform; it becomes the single source of truth for your entire Cloud-Native ecosystem. This fundamental shift means you eliminate the need for complex, layered infrastructure tools. Why? Because when everything—from microservices and operators to ephemeral serverless functions—communicates only through the extended Kubernetes API, Kubernetes itself manages all policy, messages, networking, state, and role-based-access-control. This unified control plane eliminates the friction and complexity of maintaining external service discovery, configuration layers, service inter-communications, external databases, and other networking tools. In this new paradigm, the architecture is so cohesive and Kubernetes-native that we have to introduce a new abstraction: The Service Mesh is Dead. Its core functionalities (like traffic routing, security, access control, and observability) are now handled directly by the high-performance, API-centric platform that HariKube has enabled.

Serverless — Nanoservice Layer

With a lightweight watch connector, every Custom Resource Definition (CRD) or core resource change can trigger a function running on OpenFaaS or Knative. The function layer provides event-driven business logic without requiring operators, custom APIs, or external event buses.

Developers only need a CRD and a function image.
Kubernetes acts as the event source (RBAC, Network Policies, namespaces included).
The function focuses on logic, the platform handles everything else.

Function languages: Go, Python, Node.js/TypeScript, Java, C#/.NET, Ruby, PHP, Rust.

In practice, any language that runs in a container and speaks HTTP works.

This seamless integration also extends directly into the event ecosystem. We've developed a groundbreaking tool that transforms Kubernetes events into serverless function triggers. This capability allows your business logic to react instantly and efficiently to any change within the cluster's state—from Pod failure, Custom Resource CRUD operations to ConfigMap updates—without complex external polling or sidecar injection. If you're interested in understanding the precise mechanisms behind this real-time, event-driven architecture, and how HariKube uses it to further simplify your control loop logic, please visit this page: Declarative Event Triggers for Kubernetes.

Operators — Microservice Layer

Operators remain the best way to handle stateful, long-lived, or complex business logic. With HariKube’s data fabric, operators behave like regular microservices without being bottlenecked by the database. You can think about Kubernetes as document data warehouse.

When you need reconcile loops, complex dependencies, or imperative operations — this is your layer.

Operator languages: Go, Python, Java, Node.js/TypeScript, Rust, C#/.NET.

While HariKube improves performance over single-ETCD setups, it inherits some
Kubernetes design tradeoffs.

Limitation	Description
Eventual Consistency	Updates may not be immediately visible across the system
Non-Transactional	No support for ACID transactions
Relational Logic	Complex joins or relations between data entities are not supported
Limited Data Filtering	No advanced query engine included within Kubernetes

To dive deep into the best practices, patterns, and code examples for building next-generation operators on HariKube, be sure to visit our comprehensive series of blog posts dedicated exclusively to these topics:

Kubernetes Aggregation API Layer — Traditional REST API Layer

Some use cases don’t fit into serverless or operator patterns — for example, classic REST APIs, querying, or external integrations. Here, Kubernetes’ Aggregation API layer lets you embed a custom API server directly into the Kubernetes control plane. Your custom API becomes a full featured member of Kubernetes.

Typical REST/API languages: Go, Python, Node.js/TypeScript, Java, C#/.NET, Rust, Ruby, PHP.

In practice, any language that runs in a container and speaks HTTP works.

Our approach fundamentally transforms this layer from a simple extension point into the central nervous system of the HariKube platform. If you are interested in a technical deep dive on how we utilize the Kubernetes Aggregation API to embed your custom API seamlessly, we detail the entire process in this specialized blog post: A Quick Dive into Kubernetes Operators - Custom APIs.

Strategic Impact

Current available platforms separate these domains:

Functions → Knative / OpenFaaS
Operators → Kubebuilder / controller-runtime / Operator SDK
APIs → Ingress + external apps

HariKube brings them together into a single architecture, centered on the Kubernetes API and powered by its dynamic backend. This hybrid model:

Unifies three different development paradigms on one platform. Functions, operators, and REST APIs are all exposed through the Kubernetes API. This simplifies observability, policy enforcement, and lifecycle management.
Easier horizontal scaling, multi-database routing prevents the traditional ETCD bottleneck, allowing control plane throughput to scale with workload size.
Reduces time to market, because developers can focus entirely on business logic instead of building bespoke infrastructure layers.
Cuts development costs, by eliminating the need to maintain separate stacks for serverless, operators, and APIs.
Lower learning curve, developers don’t need to learn multiple frameworks or infrastructure stacks. They can pick the layer that matches their problem domain.
Language freedom, each layer supports a wide set of languages. Teams can use the language they’re strongest in without compromising platform integration.
Consistent development workflow, via functions, operators, and APIs all use the same CRD-based declarative model.
Local-first development, because HariKube is transparent, developers can run stock/vanilla stacks locally or in ephemeral dev environments without any dependencies.

Stop Waiting: Welcome to the Platform-as-a-Service Reality

The architectural innovations we've detailed—solving the scalability crisis and unifying service design—are not roadmap promises; they are current reality. At HariKube, we've moved past the traditional concept of Kubernetes as merely a container scheduler. We have engineered the components to allow businesses to deploy the future today. This means you don't have to wait for the cloud vendors or upstream Kubernetes to catch up to the needs of modern enterprise scale.

Frankly, we're too excited to keep this under wraps any longer! The theoretical elegance of a unified, scalable Kubernetes platform is only half the story; the real power is in the implementation. We can't wait to show you exactly how HariKube transforms these concepts into deployable, functional reality, changing the way your teams build and manage cloud-native services forever.

Prerequisites: Sign Up for Open Beta Access

This platform revolution is happening now, and we want you to be a part of it. To begin leveraging HariKube's unified control plane, dynamic data layer, and truly scalable CNPaaS capabilities, the only prerequisite is signing up. Access to our Open Beta is now available! Join the early adopters who are fundamentally changing how they approach cloud-native development. Visit Open Beta invitation today to secure your spot and start building on the future of Kubernetes.

Bring Your Own Kubernetes

The true power of HariKube's architectural breakthrough is its complete infrastructure agnosticism. It doesn't matter if you're running on a hyper-scale Managed Kubernetes offering like GKE, EKS, or AKS; on a tightly controlled Self-Hosted cluster in your private data center; or on a lightweight Kind cluster on a developer's laptop. HariKube supports it all, because it is totally transparent to Kubernetes itself. This freedom of choice ensures that you can execute your architecture strategy—from local development to global, federated deployments—without ever sacrificing the enhanced scalability or unified service design that defines the future.

To successfully deploy the full HariKube platform, your existing Kubernetes cluster must first have the essential prerequisites of Cert Manager for certificate lifecycle management and Prometheus for metrics and observability.

If they are missing, install them with the commands below.

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.16.3/cert-manager.yaml
kubectl apply -f https://github.com/prometheus-operator/prometheus-operator/releases/download/v0.77.1/stripped-down-crds.yaml

The New PaaS Starts Here: HariKube Installation

The moment to stop talking about the future and start building it is now. Assuming you've completed the prerequisite step and have your beta access credentials, executing the fundamental transformation of your Kubernetes cluster requires just a few straightforward commands. By running the following sequence, you will introduce the core HariKube components, implementing a separated Kubernetes API via vCluster in the background to ensure genuine architectural separation between infrastructure management and service data, and immediately upgrade your Kubernetes environment to a highly scalable, platform-ready PaaS foundation. Get ready to witness the convergence of power and simplicity:

kubectl create namespace harikube
kubectl create secret docker-registry harikube-registry-secret \
--docker-server=registry.harikube.info \
--docker-username=<oci-user> \
--docker-password='<my$secure@password>' \
--namespace=harikube
kubectl apply -f https://harikube.info/manifests/harikube-operator-beta-v1.0.0-2.yaml
kubectl apply -f https://harikube.info/manifests/harikube-middleware-vcluster-api-beta-v1.0.0-18.yaml

Wait for all pods are running in harikube namespace.

kubectl get pod -n harikube -w

Let’s create your first topology configuration. This tells the HariKube Operator how to route data for a specific custom resource, or all records in a namespace for example. You can have as many as you want. For simplicity this example for Shirts uses SQLite database, but you can freely select any of the supported databases. For more info please visit the supported databases section in the docs.

apiVersion: harikube.info/v1
kind: TopologyConfig
metadata:
  name: topologyconfig-shirts
  namespace: harikube
spec:
  targetSecret: harikube/topology-config
  backends:
  - name: shirts
    endpoint: sqlite:///db/shirts.db?_journal=WAL&cache=shared
    customresource:
      group: stable.example.com
      kind: shirts

While configuring the full dynamic data layer is crucial, we won't deeply dive into the specifics of the topology configuration details here; for comprehensive documentation and examples, please visit the following pages: topology config and automation.

That's It! Instant Platform Upgrade

That's right—it's that simple. With HariKube successfully deployed, you have fundamentally elevated your cluster. Any service running on your existing Kubernetes and configured to communicate with the newly installed virtual Kubernetes API will immediately leverage all the features mentioned in this blog post.

You now benefit from:

Works Seamlessly on Cloud and Bare-metal
True Separation of Concerns
Enhanced Multi-Tenancy and Isolation
Independent and Massive Scaling and Configuration
Fine-Grained Resource Synchronization between host and virtual clusters
Unified Service Design

The Demo Time: Let's See It in Action

Now that the HariKube PaaS foundation is successfully laid, let's see how our unified service design simplifies creating a totally new, custom service type. We begin by applying a standard Custom Resource Definition (CRD). This simple kubectl apply command registers a completely new resource—in this case, a Shirt service—directly into the control plane. Because HariKube automatically handles the dynamic data layer and API extension, this new custom resource instantly inherits all the massive scalability and unified platform features we've discussed, making it a first-class citizen of your new PaaS.

First you have to connect to the virtual Kubernetes API. PLease install vCluster, and execute the following command.

vcluster connect harikube

vCluster simplifies the operational workflow by automatically updating your local environment.

Crucially, this Kubernetes environment serves only as your control plane and data source, containing no application Pods; it is purpose-built to serve your APIs.

kubectl get pod -A
No resources found

If you want to run your services inside the visrtual cluster instead of the host, you can do it simply by using https://harikube.info/manifests/harikube-middleware-vcluster-workload-beta-v1.0.0-18.yaml instead of the API only manifest.

Now you can apply your Custom Resource Definition to extend your Kubernetes API with your custom type.

kubectl apply -f https://raw.githubusercontent.com/kubernetes/website/main/content/en/examples/customresourcedefinition/shirt-resource-definition.yaml

Then add a few resources.

cat | kubectl apply -f - <<EOF
apiVersion: stable.example.com/v1
kind: Shirt
metadata:
  name: example1
spec:
  color: blue
  size: S
---
apiVersion: stable.example.com/v1
kind: Shirt
metadata:
  name: example2
spec:
  color: blue
  size: M
---
apiVersion: stable.example.com/v1
kind: Shirt
metadata:
  name: example3
spec:
  color: green
  size: M
EOF

Verify the resources are exists.

kubectl get shirts
NAME       COLOR   SIZE
example1   blue    S
example2   blue    M
example3   green   M

Now, let's jump back to the host Kubernetes cluster—the environment where you initially ran the installation commands.

vcluster disconnect

Verify the resources are NOT exists.

kubectl get shirts
error: the server doesn't have a resource type "shirts"

The most critical realization here is that resources like your custom Shirt object do not exist on this host cluster. They are stored exclusively on the virtual cluster's API. All instances of those application-specific resources are held securely and scalably in the external database defined in your HariKube topology configuration. The host cluster's only concern is managing the infrastructure components required to run the services, virtual API and the physical nodes, while the virtual API cluster handles your APIs nothing else. This proves the separation of concerns is not just an organizational policy, but a core architectural fact.

The Future is Yours to Build

We've shown you the architecture, demonstrated the separation, and confirmed that the future of Kubernetes—the highly scalable, unified PaaS—is not a conceptual vision, but a deployable reality right now.

The next step is yours. Stop managing complex infrastructure layers and start focusing on pure business value. Join the HariKube Open Beta today. Your platform awaits.

Introducing Serverless Kube Watch Trigger: Declarative Event Triggers for Kubernetes

Richard Kovacs — Sat, 18 Oct 2025 10:21:21 +0000

Today we’re releasing something small, simple, open-source, and surprisingly powerful: serverless-kube-watch-trigger, a Kubernetes Custom Resource Definition that turns cluster events into HTTP calls — directly and declaratively.

No glue scripts. No extra brokers. No complex controllers. Just YAML.

❓ Why We Built It

When working on HariKube, we realized a missing piece: while Kubernetes could scale and orchestrate workloads, there was no simple, built-in way to react to events without writing Go code. Operators work great for stateful loops, but not every use case needs a controller.

We wanted a native, minimal way to say:

“When this Deployment changes, send an HTTP request.”

That’s what serverless-kube-watch-trigger does.

🔌 What It Does

serverless-kube-watch-trigger listens to changes in Kubernetes resources and performs HTTP requests when those changes match defined filters.

You describe what to watch, what to send, and where to send it.

That’s all.

Here’s the full example:

apiVersion: triggers.example.com/v1
kind: HTTPTrigger
metadata:
  name: full-httptrigger-example
  namespace: default
spec:
  resource:
    apiVersion: apps/v1
    kind: Deployment
  namespaces:
    - default
    - kube-system
  labelSelectors:
    - app=frontend
    - tier=prod
  fieldSelectors:
    - metadata.name=my-deployment
  eventTypes:
    - ADDED
    - MODIFIED
    - DELETED
  eventFilter: 'ne .status.availableReplicas 0'
  concurrency: 5
  sendInitialEvents: false
  url:
    static: "https://example.com/hook"
    # template: "https://example.com/hook/{{ .metadata.name }}"
    # service:
    #   name: webhook-svc
    #   namespace: default
    #   portName: http
    #   scheme: http
    #   uri:
    #     template: "/hook/{{ .metadata.name }}"
  method: POST
  auth:
    basicAuth:
      user: ci-bot
      secretKeyRef:
        name: webhook-pass
        key: password
    tls:
      caRef:
        name: webhook-ca
        key: ca.crt
      certRef:
        name: webhook-cert
        key: tls.crt
      keyRef:
        name: webhook-cert
        key: tls.key
      insecureSkipVerify: false
  headers:
    static:
      X-Static-Token: "fixed-token-value"
      X-Cluster-ID: "cluster-001"
    template:
      X-Resource-Name: "{{ .metadata.name }}"
      X-Resource-Namespace: "{{ .metadata.namespace }}"
    fromSecretRef:
      X-Api-Key:
        name: api-secret
        key: api-key
      X-Other-Token:
        name: extra-secrets
        key: token
  body:
    contentType: application/json
    template: |
      {{ toJson . }}
    signature:
      header: X-Signature
      keySecretRef:
        name: sig-key
        key: key
      hmac:
        hashType: SHA512
  delivery:
    timeout: 30s
    retries: 5

Install the controller.

kubectl apply -f https://github.com/HariKube/serverless-kube-watch-trigger/releases/download/beta-v1.0.0-6/bundle.yaml

Configure RBAC and enable deployments for the operator.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: serverless-kube-watch-trigger-deployment-rolebinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: serverless-kube-watch-trigger-deployment-role
subjects:
- kind: ServiceAccount
  name: serverless-kube-watch-trigger-controller-manager
  namespace: serverless-kube-watch-trigger-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: serverless-kube-watch-trigger-deployment-role
rules:
- apiGroups:
  - "apps"
  resources:
  - deployments
  verbs:
  - get
  - list
  - watch

Finally apply the custom resource, and whenever a Deployment’s replica count changes, Kubernetes will POST its event as JSON to your endpoint.

⚡ How It Fits Into HariKube

Within the HariKube ecosystem, serverless-kube-watch-trigger acts as the simplest way to build event-driven systems:

It connects Kubernetes watches directly to functions or APIs.
It works out of the box with OpenFaaS, Knative, or any HTTP endpoint.
It uses no custom controller logic — just Kubernetes’ built-in reconciliation.

This aligns with HariKube’s philosophy: let Kubernetes be the platform.

Instead of bolting on more systems, reuse what already exists — RBAC, namespaces, CRDs, and event streams — and keep logic outside the cluster.

🧠 Final Thoughts

Kubernetes has grown into the universal runtime for everything from batch jobs to databases. With HariKube’s data fabric and serverless-kube-watch-trigger’s declarative event hooks, it’s now also becoming the runtime for events and automation.

Small building blocks. Clear contracts. Native integration.

That’s the direction we’re heading.

If you want to try it or contribute, the README in the repository has all the details.

Sounds interesting? Learn how HariKube unifies different service design strategies into a single hybrid architecture. [→]

That’s the message! serverless-kube-watch-trigger is a Kubernetes Custom Resource that declaratively turns cluster events (watches) into HTTP calls, enabling simple, no-code automation. This lets you connect Kubernetes resource changes directly to OpenFaaS, Knative, or any webhook, aligning with HariKube’s goal of making Kubernetes the core, event-driven platform without requiring complex operators or external brokers.

Thank you for following along! If you have questions or ideas, please share them—we’d love to hear from you.

We make Kubernetes work at enterprise scale - no more ETCD limits

Richard Kovacs — Mon, 06 Oct 2025 22:03:12 +0000

Imagine a world where Kubernetes itself becomes the platform — not just for container orchestration, but for functions, microservices, and APIs. With HariKube’s dynamic data layer, serverless function triggers, operator loops, and the Kubernetes Aggregation API layer, that world is not only possible — it’s here.

This blog post explores how HariKube unifies these layers into a single hybrid architecture, enabling developers to build nanoservices, microservices, and REST APIs on the same platform — in any language they choose.

🌐 The Big Picture: Kubernetes as a Full PaaS Runtime

For years, Kubernetes has been the foundation of modern cloud infrastructure. But while it excels at scheduling, scaling, and managing workloads, its application development experience has been fragmented:

Serverless functions live in Knative or OpenFaaS.
Operators are built with Kubebuilder or Kopf for infrastructure related developments.
REST APIs are bolted on via Ingress and separate application stacks.

HariKube changes this. By replacing ETCD with a database-agnostic backend topology and connecting Kubernetes watches to serverless runtimes, reconciliation loops, and custom APIs. It turns Kubernetes into a true Cloud Native Platform-as-a-Service.

🧱 1️⃣ HariKube — Dynamic, Database-Agnostic Data Backend

HariKube replaces ETCD with a flexible middleware that routes different resource types to different databases — MySQL, Postgres, SQLite, or others. The Kubernetes API becomes a scalable, modular data fabric, decoupled from a single store.

🧠 Kubernetes stops being just a control plane store and becomes a distributed, extensible data platform.

⚡ 2️⃣ Serverless — Nanoservice Layer

Developers only need a CRD and a function image.
Kubernetes acts as the event source (RBAC, namespaces included).
The function focuses on logic, the platform handles everything else.

👉 kubectl apply → event → function trigger → database backend → fully operational nanoservice.

Function languages: Go, Python, Node.js/TypeScript, Java, C#/.NET, Ruby, PHP, Rust.

In practice, any language that runs in a container and speaks HTTP works.

🔁 3️⃣ Operators — Microservice Layer

When you need reconcile loops, complex dependencies, or imperative operations — this is your layer.

Operator languages: Go, Python, Java, Node.js/TypeScript, Rust, C#/.NET.

🌐 4️⃣ Kubernetes Aggregation API Layer — Traditional REST API Layer

Typical REST/API languages: Go, Python, Node.js/TypeScript, Java, C#/.NET, Rust, Ruby, PHP.

In practice, any language that runs in a container and speaks HTTP works.

🧠 All Layers Combined: A Hybrid Architecture

When these layers come together:

🔥 Kubernetes + HariKube + Serverless + Operators + Aggregation API = Cloud Native PaaS

Functional need	Pattern	Layer	Typical languages
Simple event-driven logic	Function	Serverless	Go, Python, Node.js/TypeScript, Java, C#/.NET, Ruby, PHP, Rust
Stateful, multi-resource event driven logic	Operator	Controller	Go, Python, Java, Node.js/TypeScript, Rust, C#/.NET
Classic API, querying, integration	REST	Aggregation API	Go, Python, Node.js/TypeScript, Java, .NET, Rust, Ruby, PHP

🚀 Strategic Impact

Current cloud platforms separate these domains:

Functions → Knative / OpenFaaS
Operators → Kubebuilder / controller-runtime
APIs → Ingress + external apps

HariKube brings them together into a single architecture, centered on the Kubernetes API and powered by its dynamic backend. This hybrid model:

Decouples development from infrastructure, enabling faster iteration cycles. Infrastructure teams can focus on platform and scaling. Developers focus on business logic and data models. This clean split lowers cognitive overhead and onboarding time.
Unifies three different development paradigms on one platform. Functions, operators, and REST APIs are all exposed through the Kubernetes API. This simplifies observability, policy enforcement, and lifecycle management.
Easier horizontal scaling, multi-database routing prevents the traditional ETCD bottleneck, allowing control plane throughput to scale with workload size.
Improves system performance compared to single-database Kubernetes deployments, delivering higher throughput, lower latency, and reduced error rates through distributed, resource-type-specific data routing.
Simplified maintenance, each backend can be updated, scaled, or backed up independently, without cluster-wide downtime.
Built-in multi-tenancy and isolation, by assigning different backends to different resource groups or namespaces, you can enforce isolation boundaries without spinning up extra clusters.
Reduces time to market, because developers can focus entirely on business logic instead of building bespoke infrastructure layers.
Cuts development costs, by eliminating the need to maintain separate stacks for serverless, operators, and APIs.
Lower learning curve, developers don’t need to learn multiple frameworks or infrastructure stacks. They can pick the layer that matches their problem domain.
Language freedom, each layer supports a wide set of languages. Teams can use the language they’re strongest in without compromising platform integration.
Consistent development workflow, via functions, operators, and APIs all use the same CRD-based declarative model.
Local-first development, because HariKube is transparent, developers can run full stacks locally or in ephemeral dev environments without any dependencies.
Easier regulatory compliance, data placement policies make it easier to comply with GDPR, HIPAA, or financial regulations by physically separating data types and regions.
Enables new business models, platform teams can expose the underlying hybrid PaaS to internal or external customers, offering Functions, APIs, and Operators as products.
Future-proof architecture, as new backend technologies appear, HariKube can integrate them transparently without changing how developers build services.

✨ A New Platform Strategy

This is more than technical elegance — it’s a blueprint for the next generation of cloud platforms:

👉 Kubernetes as a Function + Microservice + REST API runtime

👉 HariKube as a database-agnostic, extensible data fabric

👉 Watch connectors as event-bus-free serverless triggers

👉 Controller loops as new way of microservice development

👉 Aggregation API as the native API gateway

👉 Developers using Go, Python, Node.js, Java, .NET, Rust, Ruby, PHP, or even C++ — all on the same platform.

🚀 HariKube transforms Kubernetes into a full hybrid PaaS — flexible, language-agnostic, and built for the future.

Ready to run your application in production? Learn how to prepare your Kubernetes to use HariKube as underlaying storage. →

That’s it! Imagine your own data topology and enhance your Kubernetes experience. Enjoy lower latency, higher throughput, data isolation, virtually unlimited storage, and simplified development. HariKube supports both flat and hierarchical topologies, allowing you to organize your databases like leaves on a tree.

Thank you for reading, and feel free to share your thoughts.

A Quick Dive into Kubernetes Operators - Part 4

Richard Kovacs — Sun, 28 Sep 2025 20:27:09 +0000

Before diving in, make sure you have finished part 3 of the series. →

In the previous parts of this series, we built the foundation for a Kubernetes operator, explored how to expose and manage custom resources and implememnted a custom search API. Now, let’s take the next step: implementing fully custom APIs inside Kubernetes. This allows you to present complex, aggregated, or domain-specific custom logic directly through the Kubernetes API.

Implement a Fully Customized API

Sometimes, the built-in API and operator capabilities aren't enough, especially when dealing with complex data relationships or when you need fine-grained control over the data retrieval process. In these cases, you can implement fully customized services using an aggregation API.

When Simple Solutions Fall Short

Traditional solutions like eager loading work well for one-to-one or one-to-many relationships, but they can become inefficient or unwieldy when you have:

Many-to-many relationships
Complex data transformations
Deeply nested relationships

By using an aggregation API, you decouple the frontend from the complex backend logic. This approach gives you full control over the data retrieval process, allowing you to optimize performance, reduce network chatter, and tailor the data precisely for each client's specific requirements.

In this example you'll create a cluster scope ClusterTask and a namespaced scope CustomTask service. Bring your own implementation.

{
  ApiResource: metav1.APIResource{
    Name:  "clustertasks",
    Kind:  "ClusterTask",
    Verbs: []string{"get", "list", "watch", "create", "update", "delete"},
  },
  CustomResources: []kaf.CustomResource{
    {
      CreateHandler: func(namespace, name string, w http.ResponseWriter, r *http.Request) {
        w.Header().Set("Content-Type", "application/json; charset=utf-8")
      },
      GetHandler: func(namespace, name string, w http.ResponseWriter, r *http.Request) {
        w.Header().Set("Content-Type", "application/json; charset=utf-8")
      },
      ListHandler: func(namespace, name string, w http.ResponseWriter, r *http.Request) {
        w.Header().Set("Content-Type", "application/json; charset=utf-8")
      },
      ReplaceHandler: func(namespace, name string, w http.ResponseWriter, r *http.Request) {
        w.Header().Set("Content-Type", "application/json; charset=utf-8")
      },
      DeleteHandler: func(namespace, name string, w http.ResponseWriter, r *http.Request) {
        w.Header().Set("Content-Type", "application/json; charset=utf-8")
      },
      WatchHandler: func(namespace, name string, w http.ResponseWriter, r *http.Request) {
        w.Header().Set("Content-Type", "application/json; charset=utf-8")
      },
    },
  },
},
{
  ApiResource: metav1.APIResource{
    Name:       "customtasks",
    Namespaced: true,
    Kind:       "CustomTask",
    Verbs:      []string{"get", "list", "watch", "create", "update", "delete"},
  },
  CustomResources: []kaf.CustomResource{
    {
      CreateHandler: func(namespace, name string, w http.ResponseWriter, r *http.Request) {
        w.Header().Set("Content-Type", "application/json; charset=utf-8")
      },
      GetHandler: func(namespace, name string, w http.ResponseWriter, r *http.Request) {
        w.Header().Set("Content-Type", "application/json; charset=utf-8")
      },
      ListHandler: func(namespace, name string, w http.ResponseWriter, r *http.Request) {
        w.Header().Set("Content-Type", "application/json; charset=utf-8")
      },
      ReplaceHandler: func(namespace, name string, w http.ResponseWriter, r *http.Request) {
        w.Header().Set("Content-Type", "application/json; charset=utf-8")
      },
      DeleteHandler: func(namespace, name string, w http.ResponseWriter, r *http.Request) {
        w.Header().Set("Content-Type", "application/json; charset=utf-8")
      },
      WatchHandler: func(namespace, name string, w http.ResponseWriter, r *http.Request) {
        w.Header().Set("Content-Type", "application/json; charset=utf-8")
      },
    },
  },
},

Re-deploy the application.

export IMG=controller:dev
make docker-build
../bin/kind load docker-image $IMG
make deploy
kubectl delete pod -n my-project-system -l control-plane=controller-manager

kubectl create --raw /apis/search.task.example.example.com/v1/clustertasks/foo -f clustertask.yaml
kubectl replace --raw /apis/search.task.example.example.com/v1/clustertasks/foo -f clustertask.yaml
kubectl get clustertasks
kubectl get clustertasks foo
kubectl get clustertasks -w
kubectl get clustertasks foo -w
kubectl delete clustertasks foo

kubectl create --raw /apis/search.task.example.example.com/v1/namespaces/default/customtasks/foo -f customtask.yaml
kubectl replace --raw /apis/search.task.example.example.com/v1/namespaces/default/customtasks/foo -f clustertask.yaml
kubectl get customtasks
kubectl get customtasks foo
kubectl get customtasks -w
kubectl get customtasks foo -w
kubectl delete customtasks foo

Extending Built-in Resources with Aggregated Endpoints

The next service extends the Kubernetes API by creating a new combinedtasks endpoint. This endpoint behaves like any other Kubernetes resource (get, list, and watch), but instead of returning a plain Task, it provides a richer view that includes:

The Task itself — with its metadata and spec fields (priority, deadline, details, etc.).
All related Events — collected by label selector and attached under the status.events field.

In other words, a CombinedTask is a merged resource that combines the task object and its lifecycle history.

This design is useful because developers don’t need to manually query both tasks and events to understand what’s happening. By fetching combinedtasks, you immediately see the business object (Task) together with the operational signals (Events) in one unified response.

First you have to add new permissions to fetch events at internal/apiserver/task_search.go file header.

// +kubebuilder:rbac:groups="",resources=events,verbs=get;list

You’ll continue by defining a CombinedTask resource that aggregates a Task with its related Events.

type TaskStatus struct {
    Events []corev1.Event `json:"events"`
}

type CombinedTask struct {
    metav1.TypeMeta   `json:",inline"`
    metav1.ObjectMeta `json:"metadata,omitempty"`

    Spec   examplev1.TaskSpec `json:"spec,omitempty"`
    Status TaskStatus         `json:"status,omitempty"`
}

type CombinedTaskList struct {
    metav1.TypeMeta `json:",inline"`
    metav1.ListMeta `json:"metadata,omitempty"`
    Items           []CombinedTask `json:"items"`
}

Update function signature, because new endpoint has to reach Kubernetes API.

func New(kubeClient client.Client, dynamicKubeClient *dynamic.DynamicClient, port, certFile, keyFile, dataDir string) *searchAPIServer {

Pass Kubernetes client to kaf.ServerConfig.

Server: *kaf.NewServer(kaf.ServerConfig{
  KubeClient:        kubeClient,
  DynamicKubeClient: dynamicKubeClient,

Add service implementation to kaf.ServerConfig.APIKInds.

{
  ApiResource: metav1.APIResource{
    Name:       "combinedtasks",
    Namespaced: true,
    Kind:       "CombinedTask",
    Verbs:      []string{"get", "list", "watch"},
  },
  Resources: []kaf.Resource{
    {
      CreateNew: func() (schema.GroupVersionResource, client.Object) {
        return examplev1.GroupVersion.WithResource("tasks"), &examplev1.Task{}
      },
      CreateNewList: func() (schema.GroupVersionResource, client.ObjectList) {
        return examplev1.GroupVersion.WithResource("tasklist"), &examplev1.TaskList{}
      },
      ListCallback: func(ctx context.Context, namespace, _ string, objList client.ObjectList) (any, error) {
        taskList, ok := objList.(*examplev1.TaskList)
        if !ok {
          return nil, fmt.Errorf("failed to convert tasklist for: %s", objList.GetObjectKind().GroupVersionKind().String())
        }

        combinedTasks := CombinedTaskList{
          TypeMeta: metav1.TypeMeta{
            Kind:       "CombinedTaskList",
            APIVersion: Group + "/" + Version,
          },
          ListMeta: metav1.ListMeta{
            ResourceVersion:    taskList.ResourceVersion,
            Continue:           taskList.Continue,
            RemainingItemCount: taskList.RemainingItemCount,
          },
          Items: []CombinedTask{},
        }

        rawLabelSelector := "example.example.com/task-uid=" + string(taskList.Items[0].UID)
        if len(taskList.Items) > 1 {
          taskUIDs := make([]string, 0, len(taskList.Items))
          for _, t := range taskList.Items {
            taskUIDs = append(taskUIDs, string(t.UID))
          }

          rawLabelSelector = "example.example.com/task-uid in (" + strings.Join(taskUIDs, ",") + ")"
        }

        eventLabelSelector, _ := labels.Parse(rawLabelSelector)

        events := corev1.EventList{}
        if err := kubeClient.List(ctx, &events, &client.ListOptions{
          Namespace:     namespace,
          LabelSelector: eventLabelSelector,
        }); err != nil {
          return nil, fmt.Errorf("failed to list events: %v", err)
        }

        eventsByUID := map[string][]corev1.Event{}
        for _, e := range events.Items {
          uid := e.Labels["example.example.com/task-uid"]
          if _, ok := eventsByUID[uid]; !ok {
            eventsByUID[uid] = []corev1.Event{}
          }
          eventsByUID[uid] = append(eventsByUID[uid], e)
        }

        items, err := meta.ExtractList(objList)
        if err != nil {
          return nil, fmt.Errorf("failed to extract list: %v", err)
        }

        for _, t := range items {
          task := t.(*examplev1.Task)

          ct := CombinedTask{
            TypeMeta: metav1.TypeMeta{
              Kind:       "CombinedTask",
              APIVersion: Group + "/" + Version,
            },
            ObjectMeta: task.ObjectMeta,
            Spec:       task.Spec,
            Status: TaskStatus{
              Events: eventsByUID[string(task.UID)],
            },
          }

          combinedTasks.Items = append(combinedTasks.Items, ct)
        }

        return combinedTasks, nil
      },
      WatchCallback: func(ctx context.Context, _, _ string, unstructuredObj *unstructured.Unstructured) (any, error) {
        task := examplev1.Task{}
        if err := runtime.DefaultUnstructuredConverter.FromUnstructured(unstructuredObj.Object, &task); err != nil {
          return nil, fmt.Errorf("failed to convert unstructured: %v", err)
        }

        ct := CombinedTask{
          TypeMeta: metav1.TypeMeta{
            Kind:       "CombinedTask",
            APIVersion: Group + "/" + Version,
          },
          ObjectMeta: task.ObjectMeta,
          Spec:       task.Spec,
        }

        eventLabelSelector, _ := labels.Parse("example.example.com/task-uid=" + string(task.UID))

        events := corev1.EventList{}
        if err := kubeClient.List(ctx, &events, &client.ListOptions{
          Namespace:     task.Namespace,
          LabelSelector: eventLabelSelector,
        }); err != nil {
          return nil, fmt.Errorf("failed to list events: %v", err)
        }

        ct.Status = TaskStatus{
          Events: events.Items,
        }

        return ct, nil
      },
    },
  },
},

Change API initialization at cmd/main.go file.

kubeClient, err := client.New(ctrl.GetConfigOrDie(), client.Options{
  Scheme: scheme,
})
if err != nil {
  setupLog.Error(err, "unable to create kube client")
  os.Exit(1)
}

dynamicKubeClient, err := dynamic.NewForConfig(ctrl.GetConfigOrDie())
if err != nil {
  setupLog.Error(err, "unable to create dynamic kube client")
  os.Exit(1)
}

if err := mgr.Add(apiserver.New(kubeClient, dynamicKubeClient, apiServerPort, apiServerCertFile, apiServerKeyFile, apiServerDataDir)); err != nil {
  setupLog.Error(err, "unable to add API server to manager")
  os.Exit(1)
}

To associate events with tasks and avoiding N+1 query problem, the controller must label them with the task UID. Update your controller to set index label on the created events at internal/controller/task_controller.go.

event := corev1.Event{
  ObjectMeta: ctrl.ObjectMeta{
    Name:      krand.String(40),
    Namespace: task.Namespace,
    Labels: map[string]string{
      "example.example.com/task-uid": string(task.UID),
    },
  },
...

Re-deploy the application.

make docker-build
../bin/kind load docker-image $IMG
make deploy
kubectl delete pod -n my-project-system -l control-plane=controller-manager

Create your Task object.

cat | kubectl apply -f - <<EOF
apiVersion: example.example.com/v1
kind: Task
metadata:
  name: task-sample-5
spec:
  priority: 2
  details: Sample task details
  deadline: "2025-08-19T16:52:15Z"
EOF

Validate business logic via kubectl for simplicity, fetch the combinedtasks in all namespaces.

kubectl get combinedtasks -A

NAMESPACE   NAME             AGE
default     task-sample-1   45h
default     task-sample-2    2d5h
default     task-sample-3    45h
default     task-sample-4    45h
default     task-sample-5    3s

Fetch the combinedtasks in default namespace.

kubectl get combinedtasks -n default

NAMESPACE   NAME             AGE
default     task-sample-1   45h
default     task-sample-2    2d5h
default     task-sample-3    45h
default     task-sample-4    45h
default     task-sample-5    5s

Fetch a combinedtask by name in default namespace.

kubectl get combinedtasks -n default task-sample-5

NAME            AGE
task-sample-5   8s

Validate combinedtask contains events.

kubectl get combinedtasks -n default task-sample-5 -o yaml

apiVersion: v1
items:
- apiVersion: search.task.example.example.com/v1
  kind: CombinedTask
  metadata:
    ...
    name: task-sample-5
    namespace: default
  spec:
    deadline: "2025-08-19T16:52:15Z"
    details: Sample task details for task-sample-4
    priority: 3
    taskState: Pending
  status:
    events:
    - eventTime: null
      firstTimestamp: null
      involvedObject:
        apiVersion: example.example.com/v1
        kind: Task
        name: task-sample-5
        namespace: default
        resourceVersion: "128025"
        uid: 21e694cb-4735-45a5-91c4-198b0cf01311
      lastTimestamp: null
      message: Task has been updated
      ...
      reason: TaskUpdated
      reportingComponent: ""
      reportingInstance: ""
      source: {}
      type: Normal
    - eventTime: null
      firstTimestamp: null
      involvedObject:
        apiVersion: example.example.com/v1
        kind: Task
        name: task-sample-5
        namespace: default
        resourceVersion: "128025"
        uid: 21e694cb-4735-45a5-91c4-198b0cf01311
      lastTimestamp: null
      message: Task has been created
      metadata:
        ...
      reason: TaskCreated
      reportingComponent: ""
      reportingInstance: ""
      source: {}
      type: Normal
kind: List
metadata:
  resourceVersion: "128025"

Kubernetes is a powerful platform, but its native API has limitations for complex, domain-specific logic. By using the API Aggregation Layer, you can create fully customized APIs that extend Kubernetes' functionality and address these challenges. This approach allows you to implement specialized endpoints.

Ready to run your application in production? Learn how to prepare your Kubernetes to use HariKube as underlaying storage. →

Thank you for reading, and feel free to share your thoughts.

A Quick Dive into Kubernetes Operators - Part 3

Richard Kovacs — Sun, 28 Sep 2025 20:27:00 +0000

Before diving in, make sure you have finished part 2 of the series. →

The Challenge with Kubernetes Data

Kubernetes is an incredibly powerful container orchestration platform, but it has some limitations you should be aware of. One key challenge is its limited data filtering capabilities. While it excels at managing and scaling workloads, the platform doesn't include an advanced query engine out of the box. This means you can't easily perform complex, fine-grained searches or data manipulation directly within Kubernetes, which can be a roadblock when you need to find records based on full text search for example.

Finding a Solution

Don't let this limitation discourage you! The good news is that there are many robust alternatives for advanced data filtering and full text search that integrate seamlessly with Kubernetes. These external solutions are specifically designed to handle the complex queries and data analysis that Kubernetes lacks. This blog post will explore how you can overcome this challenge, focusing on one powerful solution that will give you the flexibility and control you need to effectively filter your data.

The solution to a lack of advanced data filtering often leverages the Kubernetes API Aggregation Layer. This layer acts as a proxy, sitting in front of the core Kubernetes API server. It enables you to build and run your own custom API servers that serve custom APIs. When a client like kubectl makes a request to a registered API path, the aggregation layer intercepts it and transparently forwards it to your custom API server. This powerful design pattern allows a custom API to handle data and business logic entirely independent of Kubernetes' core database, and the perfect solution for tasks like full-text search or complex data analysis that are not natively supported.

In this tutorial, you'll create a simple implementation. Don't use this in production, this is only useful for demonstrating simplicity of Kubernetes API aggregation layer! This implementation uses a metadata indexer to find the items for a given text search, which you can then use to filter the data in the second query. The main benefit of this solution is, it works seamlessly with all the supported databases.

⚠️ This two-step process can be inefficient and slow down operations,
especially in clusters with a large number of resources. However, as
the developer of a custom API server, you have full control over the
data retrieval process. Instead of relying on a pre-retrieval filter,
you can implement a more performant solution directly within the API
server's logic. There are plenty of other options like:
 - Returning the objects directly, instead of the task-indexers only.
 - On case of SQL backend and Kubernetes API application/json
    storage-media-type, you can search directly in the database.
 - Lastly but not least, you can use custom logic to store and fetch
    these resources instead of relaying on Kubernetes API.

Prepare Server Manifests

By creating an APIService object, you tell Kubernetes to route specific API requests to your own, purpose-built API server. This server can be designed to do anything, from collecting metrics (like the Metrics Server) to providing full-text search capabilities on your data. Because the data is stored and managed by your external service, you have complete control over how it is queried, filtered, and presented. This approach offers a seamless user experience, as the custom APIs are accessed with the same tools and authentication methods as the core Kubernetes APIs, but they are powered by a solution optimized for the task.

In the first step create an APIService manifest at config/manager/task-apiservice.yaml.

apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  name: v1.search.task.example.example.com
  annotations:
    cert-manager.io/inject-ca-from: SERVICE_NAME/SERVICE_NAMESPACE
spec:
  group: search.task.example.example.com
  version: v1
  groupPriorityMinimum: 1000
  versionPriority: 10
  service:
    name: v1-search-task
    namespace: system
    port: 443
  caBundle: ""

Next create the Service related manifests at config/manager/task-search-service.yaml.

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: v1-search-task-ca
  namespace: system
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: v1-search-task-cert
  namespace: system
spec:
  secretName: v1-search-task-tls
  issuerRef:
    name: v1-search-task-ca
  commonName: v1-search-task.system.svc
  dnsNames:
  - SERVICE_NAME.SERVICE_NAMESPACE.svc
  - SERVICE_NAME.SERVICE_NAMESPACE.svc.cluster.local
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/name: my-project
    app.kubernetes.io/managed-by: kustomize
  name: v1-search-task
  namespace: system
spec:
  ports:
    - port: 443
      protocol: TCP
      targetPort: 7443
  selector:
    control-plane: controller-manager
    app.kubernetes.io/name: my-project

Then add these two manifests to the resources section in the config/manager/kustomization.yaml file.

- task-search-service.yaml
- task-apiservice.yaml

Create the Deployment patch file at config/default/manager_apiservice_patch.yaml.

- op: add
  path: /spec/template/spec/containers/0/args/-
  value: --apiserver-port=:7443

- op: add
  path: /spec/template/spec/containers/0/args/-
  value: --apiserver-data-dir=/data

- op: add
  path: /spec/template/spec/containers/0/volumeMounts/-
  value:
    mountPath: /data
    name: apiserver-data

- op: add
  path: /spec/template/spec/volumes/-
  value:
    name: apiserver-data
    emptyDir: {}

- op: add
  path: /spec/template/spec/containers/0/args/-
  value: --apiserver-cert-file=/tmp/k8s-apiserver-server/serving-certs/tls.crt

- op: add
  path: /spec/template/spec/containers/0/args/-
  value: --apiserver-key-file=/tmp/k8s-apiserver-server/serving-certs/tls.key

- op: add
  path: /spec/template/spec/containers/0/ports/-
  value:
    containerPort: 7443
    name: apiserver
    protocol: TCP

- op: add
  path: /spec/template/spec/containers/0/volumeMounts/-
  value:
    mountPath: /tmp/k8s-apiserver-server/serving-certs
    name: apiserver-certs
    readOnly: true

- op: add
  path: /spec/template/spec/volumes/-
  value:
    name: apiserver-certs
    secret:
      secretName: v1-search-task-tls

The patch contains a temporary directory. All data will lost between restarts! If you want to persist data, you have to create a Persistent Volume or set hostPath instead of emptyDir.

Add the following lines in config/default/kustomization.yaml file to the patches section.

- path: manager_apiservice_patch.yaml
  target:
    kind: Deployment

Add the lines below to the replacements section in config/default/kustomization.yaml.

- source:
    kind: Certificate
    group: cert-manager.io
    version: v1
    name: v1-search-task-cert
    fieldPath: .metadata.namespace
  targets:
    - select:
        kind: APIService
        group: apiregistration.k8s.io
        version: v1
        name: v1.search.task.example.example.com
      fieldPaths:
        - .metadata.annotations.[cert-manager.io/inject-ca-from]
      options:
        delimiter: /
        index: 0
- source:
    kind: Certificate
    group: cert-manager.io
    version: v1
    name: v1-search-task-cert
    fieldPath: .metadata.name
  targets:
    - select:
        kind: APIService
        group: apiregistration.k8s.io
        version: v1
        name: v1.search.task.example.example.com
      fieldPaths:
        - .metadata.annotations.[cert-manager.io/inject-ca-from]
      options:
        delimiter: /
        index: 1

- source:
    kind: Service
    version: v1
    name: v1-search-task
    fieldPath: .metadata.name
  targets:
    - select:
        kind: Certificate
        group: cert-manager.io
        version: v1
        name: v1-search-task-cert
      fieldPaths:
        - .spec.dnsNames.0
        - .spec.dnsNames.1
      options:
        delimiter: '.'
        index: 0
        create: true
- source:
    kind: Service
    version: v1
    name: v1-search-task
    fieldPath: .metadata.namespace
  targets:
    - select:
        kind: Certificate
        group: cert-manager.io
        version: v1
        name: v1-search-task-cert
      fieldPaths:
        - .spec.dnsNames.0
        - .spec.dnsNames.1
      options:
        delimiter: '.'
        index: 1
        create: true

Implement Aggregation Server

It is time to write the aggregation service implementation itself. Create the file internal/apiserver/task_search.go with the following content.
The solution includes a tool that indexes data to make searching for task details easier and also has a raw endpoint to filter results.

package apiserver

import (
    "context"
    "errors"
    "fmt"
    "net/http"
    "strings"

    bleve "github.com/blevesearch/bleve/v2"
    kaf "github.com/mhmxs/kubernetes-aggregator-framework/pkg/framework"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    logf "sigs.k8s.io/controller-runtime/pkg/log"
)

const (
    Group   = "search.task.example.example.com"
    Version = "v1"
)

var (
    tasksearchlog = logf.Log.WithName("task-search")

    Indexer bleve.Index
)

type Index struct {
    Namespace  string `json:"namespace"`
    Name       string `json:"name"`
    Generation int64  `json:"generation"`
    Details    string `json:"details"`
}

func New(port, certFile, keyFile, dataDir string) *searchAPIServer {
    sas := searchAPIServer{
        Server: *kaf.NewServer(kaf.ServerConfig{
            Port:     port,
            CertFile: certFile,
            KeyFile:  keyFile,
            Group:    Group,
            Version:  Version,
            APIKinds: []kaf.APIKind{
                {
                    ApiResource: metav1.APIResource{
                        Name:  "details",
                        Verbs: []string{"get"},
                    },
                    RawEndpoints: map[string]http.HandlerFunc{
                        "": func(w http.ResponseWriter, r *http.Request) {
                            q := r.URL.Query().Get("q")
                            if q == "" {
                                w.WriteHeader(http.StatusBadRequest)
                                w.Write([]byte("missing query parameter 'q'"))
                                return
                            }

                            query := bleve.NewQueryStringQuery(q)
                            searchResult, err := Indexer.Search(bleve.NewSearchRequest(query))
                            if err != nil {
                                w.WriteHeader(http.StatusInternalServerError)
                                w.Write([]byte(err.Error()))
                                return
                            }

                            if len(searchResult.Hits) > 100 {
                                w.WriteHeader(http.StatusBadRequest)
                                w.Write([]byte("too many results, please narrow down your query"))
                                return
                            }

                            indexes := make([]string, 0, len(searchResult.Hits))
                            for _, hit := range searchResult.Hits {
                                indexes = append(indexes, hit.ID)
                            }

                            w.WriteHeader(http.StatusOK)
                            w.Write([]byte(strings.Join(indexes, ",")))
                        },
                    },
                },
            },
        }),
        dataDir: dataDir,
    }

    return &sas
}

type searchAPIServer struct {
    kaf.Server
    dataDir string
}

func (s *searchAPIServer) Start(ctx context.Context) (err error) {
    docMapping := bleve.NewDocumentMapping()
    docMapping.AddFieldMappingsAt("details", bleve.NewTextFieldMapping())
    docMapping.AddFieldMappingsAt("namespace", bleve.NewTextFieldMapping())

    mapping := bleve.NewIndexMapping()
    mapping.AddDocumentMapping("mapping", docMapping)
    mapping.DefaultField = "details"

    Indexer, err = bleve.New(s.dataDir, mapping)
    if err != nil {
        if errors.Is(err, bleve.ErrorIndexPathExists) {
            Indexer, err = bleve.Open(s.dataDir)
            if err != nil {
                return fmt.Errorf("failed to open index: %w", err)
            }
        } else {
            return fmt.Errorf("failed to create index: %w", err)
        }
    }

    srvErr := s.Server.Start(ctx)

    indxErr := Indexer.Close()
    if indxErr != nil {
        tasksearchlog.Error(indxErr, "Failed to close indexer")
    }

    return errors.Join(srvErr, indxErr)
}

Add new flags to the cmd/main.go file before the flag.Parse() call.

var apiServerPort string
var apiServerDataDir string
var apiServerCertFile string
var apiServerKeyFile string
flag.StringVar(&apiServerPort, "apiserver-port", ":7443", "The port the API server serves at. Default is 7443.")
flag.StringVar(&apiServerDataDir, "apiserver-data-dir", "./", "The data directory for the API server to use.")
flag.StringVar(&apiServerCertFile, "apiserver-cert-file", "", "The TLS cert file for the API server to use.")
flag.StringVar(&apiServerKeyFile, "apiserver-key-file", "", "The TLS key file for the API server to use.")

Initialize API server in the same file before the mgr.Start(ctrl.SetupSignalHandler()) call.

if err := mgr.Add(apiserver.New(apiServerPort, apiServerCertFile, apiServerKeyFile, apiServerDataDir)); err != nil {
  setupLog.Error(err, "unable to add API server to manager")
  os.Exit(1)
}

Update your mutation webhook in Default(context.Context, runtime.Object) error function at internal/webhook/v1/task_webhook.go to index resources.

lastGen := task.Generation - 10
if !task.DeletionTimestamp.IsZero() {
  lastGen = task.Generation
} else if task.Spec.Details != "" {
  indexKey := fmt.Sprintf("%s.%s.%d", task.Namespace, task.Name, task.Generation+1)
  task.Labels["example.example.com/task-indexer"] = indexKey
  if err := apiserver.Indexer.Index(indexKey, apiserver.Index{
    Namespace:  task.Namespace,
    Name:       task.Name,
    Generation: task.Generation + 1,
    Details:    task.Spec.Details,
  }); err != nil {
    return fmt.Errorf("failed to index task details: %w", err)
  }
}

if lastGen > 0 {
  cleanupQuery := bleve.NewQueryStringQuery(fmt.Sprintf("+namespace:%s +name:%s +generation:<=%d", task.Namespace, task.Name, lastGen))
  searchResult, err := apiserver.Indexer.Search(bleve.NewSearchRequest(cleanupQuery))
  if err != nil {
    return fmt.Errorf("failed to compact task details: %w", err)
  }
  batch := apiserver.Indexer.NewBatch()
  for _, hit := range searchResult.Hits {
    batch.Delete(hit.ID)
  }
  if err := apiserver.Indexer.Batch(batch); err != nil {
    return fmt.Errorf("failed to compact task details: %w", err)
  }
}

Update dependencies of the service.

go mod tidy

Re-deploy the application.

export IMG=controller:dev
make docker-build
../bin/kind load docker-image $IMG
make deploy
kubectl delete pod -n my-project-system -l control-plane=controller-manager

Create your Task object.

cat | kubectl apply -f - <<EOF
apiVersion: example.example.com/v1
kind: Task
metadata:
  name: task-sample-4
spec:
  priority: 4
  details: Sample task details for task-sample-4
  deadline: "2025-08-19T16:52:15Z"
EOF

Validate business logic by filtering tasks via aggregation api.

kubectl get tasks --selector "example.example.com/task-indexer in ($(kubectl get --raw /apis/search.task.example.example.com/v1/details?q='%2Bdetails:task-sample-4 %2Bnamespace:default'))"

NAME            PRIORITY   DEADLINE               TASKSTATE
task-sample-4   4          2025-08-19T16:52:15Z   Pending

You can read more about how to query the indexer in the official docs.

While Kubernetes excels at orchestration, its native data filtering is limited. The Kubernetes API Aggregation Layer provides a powerful solution by allowing you to create custom API servers that sit in front of the core API. This gives you the ability to implement a purpose-built data search engine, such as the one in this post, which leverages a dedicated indexer to perform complex queries like full-text searches.

Ready for the next step? Learn how to implement fully customized Aggregation API. →

Thank you for reading, and feel free to share your thoughts.

A Quick Dive into Kubernetes Operators - Part 2

Richard Kovacs — Sun, 28 Sep 2025 20:26:52 +0000

Before diving in, make sure you have finished part 1 of the series. →

Implement Your Business Logic: The Controller

Now that you've completed the first part of this series, you're ready to implement your core business logic within the controller. Your kubebuilder command has already generated a controller stub for you; all you need to do is edit the internal/controller/task_controller.go file. In this section, you'll learn how to react to different events within your cluster and create new events in Kubernetes to signal important state changes.

package controller

import (
    "context"

    corev1 "k8s.io/api/core/v1"
    apierrors "k8s.io/apimachinery/pkg/api/errors"
    "k8s.io/apimachinery/pkg/runtime"
    krand "k8s.io/apimachinery/pkg/util/rand"
    ctrl "sigs.k8s.io/controller-runtime"
    "sigs.k8s.io/controller-runtime/pkg/client"
    "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"

    examplev1 "example.com/my-project/api/v1"
)

// TaskReconciler reconciles a Task object
type TaskReconciler struct {
    client.Client
    Scheme *runtime.Scheme
}

// +kubebuilder:rbac:groups=example.example.com,resources=tasks,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=example.example.com,resources=tasks/status,verbs=get;update;patch
// +kubebuilder:rbac:groups=example.example.com,resources=tasks/finalizers,verbs=update

// +kubebuilder:rbac:groups="",resources=events,verbs=create

// Reconcile is part of the main kubernetes reconciliation loop which aims to
// move the current state of the cluster closer to the desired state.
func (r *TaskReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    task := examplev1.Task{}
    if err := r.Get(ctx, req.NamespacedName, &task); err != nil {
        if apierrors.IsNotFound(err) {
            return ctrl.Result{}, nil
        }

        return ctrl.Result{}, err
    }

    event := corev1.Event{
        ObjectMeta: ctrl.ObjectMeta{
            Name:      krand.String(40),
            Namespace: task.Namespace,
        },
        InvolvedObject: corev1.ObjectReference{
            Kind:            task.Kind,
            Name:            task.Name,
            Namespace:       task.Namespace,
            UID:             task.UID,
            APIVersion:      examplev1.GroupVersion.String(),
            ResourceVersion: task.ResourceVersion,
        },
    }

    switch {
    case task.Generation == 1:
        event.Reason = "TaskCreated"
        event.Message = "Task has been created"
        event.Type = corev1.EventTypeNormal
    case task.DeletionTimestamp.IsZero():
        event.Reason = "TaskUpdated"
        event.Message = "Task has been updated"
        event.Type = corev1.EventTypeNormal
    case !controllerutil.ContainsFinalizer(&task, "example.example.com/finalizer"):
        return ctrl.Result{}, nil
    default:
        event.Reason = "TaskDeleted"
        event.Message = "Task has been deleted"
        event.Type = corev1.EventTypeWarning

        controllerutil.RemoveFinalizer(&task, "example.example.com/finalizer")
        if err := r.Update(ctx, &task); err != nil {
            return ctrl.Result{}, err
        }
    }

    if err := r.Create(ctx, &event); err != nil {
        return ctrl.Result{}, err
    }

    return ctrl.Result{}, nil
}

// SetupWithManager sets up the controller with the Manager.
func (r *TaskReconciler) SetupWithManager(mgr ctrl.Manager) error {
    return ctrl.NewControllerManagedBy(mgr).
        For(&examplev1.Task{}).
        Named("task").
        Complete(r)
}

Once you finished editing files, you can build your OCI image by executing the following command.

export IMG=controller:dev
make docker-build

In the next step you have to make the image available on Kubernetes cluster.

../bin/kind load docker-image $IMG

And finally deploy the application to the cluster.

make deploy
kubectl delete pod -n my-project-system -l control-plane=controller-manager

Create your Task, update it and finally delete the resource.

cat | kubectl apply -f - <<EOF
apiVersion: example.example.com/v1
kind: Task
metadata:
  name: task-sample-2
  labels:
    example.example.com/priority: "2"
    example.example.com/deadline: "1755615135"
  finalizers:
  - example.example.com/finalizer
spec:
  priority: 2
  details: Sample task details
  deadline: "2025-08-19T16:52:15Z"
EOF
kubectl patch tasks task-sample-2 --type='json' -p='[{"op":"replace", "path":"/spec/taskState", "value":"Finished"}]'
kubectl delete tasks task-sample-2

Validate business logic by fetching events of the Task object.

kubectl get events \
--field-selector involvedObject.kind=Task,involvedObject.name=task-sample-2

LAST SEEN   TYPE     REASON        OBJECT               MESSAGE
<unknown>   Normal    TaskCreated   task/task-sample-2   Task has been created
<unknown>   Normal    TaskUpdated   task/task-sample-2   Task has been updated
<unknown>   Warning   TaskDeleted   task/task-sample-2   Task has been deleted

Awesome! Now that you've finished your first simple controller, you've seen how to implement the core logic of an Operator. While this was just a basic example, it's a huge first step and the foundation for building more complex applications. You're now ready to move on to more advanced topics, like validating or mutation your resources.

Custom Validation of Resources via Webhook

While the Kubernetes API has built-in validators for its core and custom resources, these are often not enough to enforce the complex business logic and custom constraints required by your applications, which is why you sometimes need to write a custom validator.

../bin/kubebuilder create webhook --group example --version v1 --kind Task --programmatic-validation --defaulting

Now you can implement your custom validation at internal/webhook/v1/task_webhook.go file.

// ValidateCreate implements webhook.CustomValidator so a webhook will be registered for the type Task.
func (v *TaskCustomValidator) ValidateCreate(_ context.Context, obj runtime.Object) (admission.Warnings, error) {
    task, ok := obj.(*examplev1.Task)
    if !ok {
        return nil, fmt.Errorf("expected a Task object but got %T", obj)
    }

    // TODO(user): fill in your validation logic upon object creation.

    return nil, nil
}

// ValidateUpdate implements webhook.CustomValidator so a webhook will be registered for the type Task.
func (v *TaskCustomValidator) ValidateUpdate(_ context.Context, oldObj, newObj runtime.Object) (admission.Warnings, error) {
    task, ok := newObj.(*examplev1.Task)
    if !ok {
        return nil, fmt.Errorf("expected a Task object for the newObj but got %T", newObj)
    }

    // TODO(user): fill in your validation logic upon object update.

    return nil, nil
}

// ValidateDelete implements webhook.CustomValidator so a webhook will be registered for the type Task.
func (v *TaskCustomValidator) ValidateDelete(ctx context.Context, obj runtime.Object) (admission.Warnings, error) {
    task, ok := obj.(*examplev1.Task)
    if !ok {
        return nil, fmt.Errorf("expected a Task object but got %T", obj)
    }

    // TODO(user): fill in your validation logic upon object deletion.

    return nil, nil
}

Mutating Resources via Webhook

A mutation webhook is a powerful Kubernetes mechanism for automatically changing or injecting resources before they are persisted, perfect for adding default values or make any conversion before they are written to the database.

Open the same internal/webhook/v1/task_webhook.go file to implement your mutation logic.

// Default implements webhook.CustomDefaulter so a webhook will be registered for the Kind Task.
func (d *TaskCustomDefaulter) Default(_ context.Context, obj runtime.Object) error {
    task, ok := obj.(*examplev1.Task)
    if !ok {
        return fmt.Errorf("expected an Task object but got %T", obj)
    }

    if task.Labels == nil {
        task.Labels = make(map[string]string)
    }

    task.Labels["example.example.com/priority"] = strconv.Itoa(int(task.Spec.Priority))
    task.Labels["example.example.com/deadline"] = strconv.Itoa(int(task.Spec.Deadline.Unix()))

    if task.Generation == 0 {
        controllerutil.AddFinalizer(task, "example.example.com/finalizer")
    }

    return nil
}

If you recall, the Task custom resource we created earlier included labels for filtering and a finalizer to control object deletion. Now, with this simple mutation webhook, you can automate the process of adding these fields, so users don't have to.

Deploying Webhooks

First you have to deploy cert-manager to enable automatic certificate generation of the webhooks.

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.16.3/cert-manager.yaml

Ucomment the following lines in config/crd/kustomization.yaml file.

configurations:
- kustomizeconfig.yaml

Ucomment the following lines in config/default/kustomization.yaml file.

- ../certmanager

replacements:

- source: # Uncomment the following block if you have any webhook
    kind: Service
    version: v1
    name: webhook-service
    fieldPath: .metadata.name # Name of the service
  targets:
    - select:
        kind: Certificate
        group: cert-manager.io
        version: v1
        name: serving-cert
      fieldPaths:
        - .spec.dnsNames.0
        - .spec.dnsNames.1
      options:
        delimiter: '.'
        index: 0
        create: true
- source:
    kind: Service
    version: v1
    name: webhook-service
    fieldPath: .metadata.namespace # Namespace of the service
  targets:
    - select:
        kind: Certificate
        group: cert-manager.io
        version: v1
        name: serving-cert
      fieldPaths:
        - .spec.dnsNames.0
        - .spec.dnsNames.1
      options:
        delimiter: '.'
        index: 1
        create: true

- source: # Uncomment the following block if you have a ValidatingWebhook (--programmatic-validation)
    kind: Certificate
    group: cert-manager.io
    version: v1
    name: serving-cert # This name should match the one in certificate.yaml
    fieldPath: .metadata.namespace # Namespace of the certificate CR
  targets:
    - select:
        kind: ValidatingWebhookConfiguration
      fieldPaths:
        - .metadata.annotations.[cert-manager.io/inject-ca-from]
      options:
        delimiter: '/'
        index: 0
        create: true
- source:
    kind: Certificate
    group: cert-manager.io
    version: v1
    name: serving-cert
    fieldPath: .metadata.name
  targets:
    - select:
        kind: ValidatingWebhookConfiguration
      fieldPaths:
        - .metadata.annotations.[cert-manager.io/inject-ca-from]
      options:
        delimiter: '/'
        index: 1
        create: true

- source: # Uncomment the following block if you have a DefaultingWebhook (--defaulting )
    kind: Certificate
    group: cert-manager.io
    version: v1
    name: serving-cert
    fieldPath: .metadata.namespace # Namespace of the certificate CR
  targets:
    - select:
        kind: MutatingWebhookConfiguration
      fieldPaths:
        - .metadata.annotations.[cert-manager.io/inject-ca-from]
      options:
        delimiter: '/'
        index: 0
        create: true
- source:
    kind: Certificate
    group: cert-manager.io
    version: v1
    name: serving-cert
    fieldPath: .metadata.name
  targets:
    - select:
        kind: MutatingWebhookConfiguration
      fieldPaths:
        - .metadata.annotations.[cert-manager.io/inject-ca-from]
      options:
        delimiter: '/'
        index: 1
        create: true

Re-deploy the application.

make docker-build
../bin/kind load docker-image $IMG
make deploy
kubectl delete pod -n my-project-system -l control-plane=controller-manager

Create your Task without metadata.

cat | kubectl apply -f - <<EOF
apiVersion: example.example.com/v1
kind: Task
metadata:
  name: task-sample-3
spec:
  priority: 5
  details: Sample task details
  deadline: "2025-08-19T16:52:15Z"
EOF

Validate mutation webhook by fetching details of the Task object.

kubectl get tasks task-sample-3 -o yaml

apiVersion: example.example.com/v1
kind: Task
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"example.example.com/v1","kind":"Task","metadata":{"annotations":{},"name":"task-sample-3","namespace":"default"},"spec":{"deadline":"2025-08-19T16:52:15Z","details":"Sample task details","priority":5}}
  creationTimestamp: "2025-08-19T17:02:47Z"
  finalizers:
  - example.example.com/finalizer
  generation: 1
  labels:
    example.example.com/deadline: "1755622335"
    example.example.com/priority: "5"
  name: task-sample-3
  namespace: default
  resourceVersion: "8867"
  uid: 7b4649e5-4f0d-4929-880a-516ede0bdde6
spec:
  deadline: "2025-08-19T16:52:15Z"
  details: Sample task details
  priority: 5
  taskState: Pending

The controller handles the core business logic, reconciling the desired state, while webhooks provide a way to enforce rules and automatically modify resources. This combination of reconciliation and admission control is what truly elevates an operator from a simple manager to a robust, self-managing application.

Ready for the next step? Learn how to implement advanced data filtering in controller. →

Thank you for reading, and feel free to share your thoughts.

A Quick Dive into Kubernetes Operators - Part 1

Richard Kovacs — Sun, 28 Sep 2025 20:26:44 +0000

What is Kubernetes and Why It's So Popular

Kubernetes is an open-source platform for automating the deployment, scaling, and management of containerized applications. It groups containers that make up an application into logical units for easy management and discovery. The popularity of Kubernetes stems from its ability to solve many of the challenges associated with managing modern, distributed applications. It provides a standardized way to handle container orchestration, ensuring applications are resilient, scalable, and portable across different cloud and on-premise environments. By abstracting the underlying infrastructure, Kubernetes allows developers to focus on writing code rather than worrying about operational complexities.

The Desired State and Controller Design Pattern

At the heart of Kubernetes is the concept of a desired state. This is a declarative model where you define the state you want your applications to be in, and Kubernetes works tirelessly to achieve and maintain that state. For example, you can declare that you want three replicas of a web server pod to be running at all times. A controller is a key component that implements this pattern. It's a control loop that continuously monitors the current state of the cluster and compares it to the desired state specified in the cluster's configuration. If there's a discrepancy (e.g., a pod crashes, so there are now only two replicas), the controller takes action to reconcile the two states (e.g., it starts a new pod). This design pattern is a fundamental reason for Kubernetes' self-healing capabilities.

Custom Resources and Operators

While Kubernetes has built-in resources like Pods, Services, and Deployments, it might not always have the perfect primitive for your specific application. This is where Custom Resources (CRs) come in. CRs are API extensions that allow you to define your own types of objects within the Kubernetes API. For example, if you're running a database, you could create a Database custom resource with fields for version, size, and user credentials.

An Operator is a specific type of controller that manages these custom resources. Operators encode the operational knowledge required to manage a complex application, like a database or a message queue. They extend the Kubernetes API by adding a new API type (the Custom Resource) and a controller that watches and manages that API type.

Features Kubernetes Gives to Run Operators

Kubernetes provides a rich set of features that empower operators to extend its native functionality and manage complex applications. These features are the building blocks that an operator uses to achieve its desired state.

Secrets and ConfigMaps: Operators rely on Secrets and ConfigMaps to manage sensitive data (passwords, API keys) and configuration settings. They can watch for changes to these resources and automatically reconfigure the application as needed, without requiring a redeployment.
- Role-Based Access Control (RBAC): This is a critical security feature that allows an operator to be granted fine-grained permissions to interact with specific resources. An operator needs an RBAC role to create, update, or delete resources. It also needs permission to manage its own Custom Resources. This principle of least privilege ensures that an operator can only perform the actions it needs to, limiting its blast radius in a security breach. For a human user to interact with the Kubernetes cluster using a tool like kubectl, they must first authenticate. The Kubernetes API server does not manage user accounts internally. Instead, it relies on external identity providers, such as:
- Client Certificates: A user's certificate is signed by the cluster's Certificate Authority (CA), identifying them as a specific user.
- Authentication Webhooks: The API server can be configured to use an external service (a webhook) to validate a bearer token, often from an Identity and Access Management (IAM) solution like Azure AD or Google IAM.
- OpenID Connect (OIDC): A popular standard that allows users to authenticate with an external identity provider and receive a token that the Kubernetes API server can validate.
Admission Controllers: Admission controllers are plugins that intercept requests to the Kubernetes API server before they are persisted. Operators can use Validating and Mutating Admission Webhooks to enforce policies on their custom resources. For example, a webhook could ensure that a user-provided database password meets certain complexity requirements.
Aggregation Layer: For more complex use cases, the Kubernetes Aggregation API allows an operator to act as an extension API server. This enables the operator to serve its own API, making its custom resources appear as if they were a native part of the Kubernetes API, complete with full support from kubectl and other standard tools.
Network Policies: Operators can define and manage Network Policies to control the traffic flow between the application's components. This allows an operator to create a secure network boundary, ensuring that only necessary communication is allowed (e.g., only the database pod can accept connections from the application pod).
Ingress/Egress Controllers: An operator for a web application can manage Ingress and Egress resources to expose the application to the outside world and control outbound traffic. This allows the operator to automate the configuration of routing, load balancing, and TLS termination.
Service Mesh Integration: For advanced networking, operators can integrate with a service mesh like Istio or Linkerd. An operator can automatically configure and manage the service mesh components for its application, enabling features like canary deployments, A/B testing, and fine-grained traffic shifting, which would be difficult to manage manually.
Cloud Native Landscape: The cloud-native landscape is a massive and constantly growing ecosystem of projects and products, with over a thousand entries in the official Cloud Native Computing Foundation (CNCF) landscape alone. The sheer breadth of this community-driven effort is one of its greatest strengths, as it ensures that there is a purpose-built tool for almost any operational challenge you might encounter. Some popular third-Party tools:
- Prometheus: An industry-standard for monitoring, Prometheus provides a powerful time-series database and a query language (PromQL) to collect and analyze metrics from applications and infrastructure.
- Istio: A feature-rich service mesh, Istio provides a transparent and automated way to manage traffic, security, and observability across microservices without requiring any code changes to the applications themselves.
- Argo CD: A popular GitOps tool for Kubernetes, Argo CD automates the deployment of applications by continuously monitoring Git repositories and synchronizing the desired state with the cluster.
- Harbor: An open-source, cloud-native container registry that secures images with vulnerability scanning, image signing, and policy checks.
- Tekton: A powerful and flexible open-source framework for creating CI/CD systems, allowing developers to build, test, and deploy applications across various cloud platforms.
- Falco: A cloud-native security tool that detects and alerts on suspicious behaviors in your cluster, such as a shell running in a container or a process spawning in an unusual location.

The Downside of Kubernetes Operators

However, custom controllers face significant challenges when handling large volumes of data. Kubernetes relies on ETCD for all data storage, which limits scalability, flexibility, and performance for complex or high-volume workloads. What are the main issues?

ETCD is designed for configuration data, and cluster size is limited.
ETCD’s consistency model means all members are full replicas, and Kubernetes only connects to the leader.
ETCD cannot filter data at the storage level; the Kubernetes API server must fetch all records and filter them client-side.
With a single ETCD instance, both cluster configuration and custom data are mixed. High-load custom services can block or delay normal cluster operations, risking overall cluster health.
Kubernetes is multi-tenant with isolation methods, but ultimately all data is stored in a single database. A database outage affects the entire cluster, and a malicious attacker could potentially access everything.

These are some of the reasons why custom resources and controllers are used to solve infrastructure problems, and why high-volume data applications were often avoided ... in the past.

While Kubernetes provides powerful APIs for managing the state of applications, it's important to understand the inherent limitations of its architectural design.

Limitation	Description
Eventual Consistency	Updates may not be immediately visible across the system
Non-Transactional	No support for ACID transactions
Relational Logic	Complex joins or relations between data entities are not supported
Limited Data Filtering	No advanced query engine included within Kubernetes

💡 Don't worry, the Kubernetes API aggregation layer can help overcome the limitations of the core API server by allowing you to extend the API with custom APIs that are served by a separate backend, or extension API server. This setup enables you to implement specific logic and capabilities that aren't available in the core API.

Turn Kubernetes into a Real Platform-as-a-Service

It’s time to meet HariKube. HariKube is a middleware that transparently distributes database load across multiple vendor-agnostic databases, delivering low latency, high throughput, and a true cloud-native development experience. It achieves exceptional performance through data distribution and optimized database routing. By offloading resource-intensive workloads from ETCD, HariKube ensures consistent responsiveness and operational efficiency at scale. It enables strict data separation across namespaces, resource types, or services—helping organizations meet security and compliance requirements without sacrificing scalability or performance. Additionally, HariKube simplifies developer workflows by abstracting infrastructure complexity. Developers can focus on data structures and business logic while the platform handles data routing and storage. HariKube is fully transparent; Kubernetes does not notice it is not communicating with an ETCD instance. You can use vanilla Kubernetes for development with limited data, and deploy HariKube in production to handle large datasets and distribute data across multiple backends.

HariKube supports multiple backends, each with different capabilities for data access and filtering. The table below outlines which storage engines are compatible and whether they support storage-level filtering for efficient querying. Find full list of databases here.

Craft Your First Operator Based Microservice

You can develop operators in various programming languages, but some have more mature frameworks than others. The most popular language for operator development is Go, primarily due to its strong type system and the existence of robust frameworks. Other languages with operator development capabilities include C#, Python, Java, Rust, Javascript and increasing. However, the Go-based tools are generally considered the most mature and widely adopted within the Kubernetes community.

In this example you would use Kubebuilder for development and Kind for testing. First of all please download dependencies.

mkdir -p bin
curl -L https://github.com/kubernetes-sigs/kubebuilder/releases/download/v4.6.0/kubebuilder_linux_amd64 -o ./bin/kubebuilder
chmod +x ./bin/kubebuilder
curl -Lo ./bin/kind https://kind.sigs.k8s.io/dl/latest/kind-linux-amd64
chmod +x ./bin/kind

In the next step you would initialize your operator.

mkdir my-project
cd my-project
../bin/kubebuilder init --domain example.com --repo example.com/my-project

The project structure looks like:

.github # Integrated CI with Github
Dockerfile # Ready to run image builder
Makefile # Make targets (run, vet, lint, test, test-e2e, generate, ...)
PROJECT # Operator definition
config # Kubernetes manifests
test # E2E test framework

At this point you should design your domain, a simple ToDo application in our case.

../bin/kubebuilder create api --group example --version v1 --kind Task --resource=true --controller=true

Edit the custom resource definition file at api/v1/task_types.go.

package v1

import (
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

type TaskState string

const (
    TaskStatePending  TaskState = "Pending"
    TaskStateFinished TaskState = "Finished"
)

// TaskSpec defines the desired state of Task.
type TaskSpec struct {
    // +kubebuilder:validation:Required
    // +kubebuilder:validation:Minimum=0
    // +kubebuilder:validation:Maximum=10
    Priority int `json:"priority,omitempty"`
    // +kubebuilder:validation:Optional
    Details string `json:"details,omitempty"`
    // +kubebuilder:validation:Required
    Deadline metav1.Time `json:"deadline,omitempty"`
    // +kubebuilder:default="Pending"
    // +kubebuilder:validation:Enum=Pending;Finished
    TaskState TaskState `json:"taskState,omitempty"`
}

// TaskStatus defines the observed state of Task.
type TaskStatus struct {
}

// +kubebuilder:object:root=true
// +kubebuilder:subresource:status
// +kubebuilder:printcolumn:name="Priority",type="string",JSONPath=".spec.priority"
// +kubebuilder:printcolumn:name="DeadLine",type="string",JSONPath=".spec.deadline"
// +kubebuilder:printcolumn:name="TaskState",type="string",JSONPath=".spec.taskState"
// +kubebuilder:selectablefield:JSONPath=.spec.priority
// +kubebuilder:selectablefield:JSONPath=.spec.taskState

// Task is the Schema for the tasks API.
type Task struct {
    metav1.TypeMeta   `json:",inline"`
    metav1.ObjectMeta `json:"metadata,omitempty"`

    Spec   TaskSpec   `json:"spec,omitempty"`
    Status TaskStatus `json:"status,omitempty"`
}

// +kubebuilder:object:root=true

// TaskList contains a list of Task.
type TaskList struct {
    metav1.TypeMeta `json:",inline"`
    metav1.ListMeta `json:"metadata,omitempty"`
    Items           []Task `json:"items"`
}

func init() {
    SchemeBuilder.Register(&Task{}, &TaskList{})
}

Once you finished editing file, you can generate all the Kubernetes manifests and source codes by executing the following command.

make manifests generate

Start your Kubernetes cluster.

../bin/kind create cluster

Deploy your custom API into Kubernetes.

make install

Create your very first Task.

cat | kubectl apply -f - <<EOF
apiVersion: example.example.com/v1
kind: Task
metadata:
  name: task-sample
  labels:
    example.example.com/priority: "3"
    example.example.com/deadline: "1755587875"
spec:
  priority: 3
  details: Sample task details
  deadline: "2025-08-19T09:17:55Z"
EOF

You can filter the objects you want to interact with in Kubernetes by using field selectors and label selectors. These are powerful mechanisms that allow you to narrow down your search from the entire cluster to a specific subset of resources. Label selectors are the more common and powerful of the two. A label is a key-value pair attached to a resource, like example.example.com/priority=3 or example.example.com/deadline>1755587874. Field selectors, on the other hand, let you filter resources based on the values of their resource fields. In this example spec.priority and spec.taskState are the pre-defined field selectors. While useful, field selectors are more limited in their filtering capabilities compared to label selectors.

You can test selectors via kubectl, but almost all the popular programming languages have existing Kubernetes client library, so it is up to you how you connect to the cluster API.

kubectl get tasks \
--selector "example.example.com/deadline>1755587874" \
--selector "example.example.com/priority=3" \
--field-selector "spec.priority=3" \
--field-selector "spec.taskState=Pending"

NAME          PRIORITY   DEADLINE               TASKSTATE
task-sample   3          2025-08-19T09:17:55Z   Pending

The Operator pattern turns Kubernetes into a powerful platform, but its reliance on etcd can create a performance bottleneck for data-heavy applications. The solution? Decouple the data. With tools like HariKube transparently offloading custom resource data, you can build scalable, high-performance services that leverage the full power of Kubernetes without compromising on speed or stability. It’s the future of cloud-native development.

Ready for the next step? Learn how to write your custom business logic via controller. →

Thank you for reading, and feel free to share your thoughts.

My pro tips for tech people with and without ADHD ;)

Richard Kovacs — Wed, 06 Dec 2023 10:10:58 +0000

or anybody else on the spectrum.

My main reason for this post, is giving back to you. Your reaction on my previous post about Coding and ADHD should be best friends just exploded into my face (in a good sense). Thank you for all the support, shares, follows and everything, I appreciate it from the bottom of my hearth. But it's enough of the tears, let's do it XD

If it is too long to read: Find the place, who wants you and not your time! Work on your interest! Be super proactive, and find your path instead of being a follower.

Have a nice and proactive day! You are awesome!
Ricsi

ps:

Disclaimer[0]: First of all, my experiences are mines. They are relatives, limited and very "western" a.k.a. first worldish. #ALLLIFEMATTERS

Disclaimer[1]: I'm not a professional therapist or health care expert. Self-diagnosis should be risky, please ask for professional help!

My brain with ADHD has different Dopamine paths compare to Neurotypical persons. I have challenges to remember things like meetings. Sometimes it is hard to sit at one position, or simply I feel I have to write a post about my pro tips, haha. Hard to predict which activity earns my focus. And I have full consciousness of my managers and teammates would read this, I hope they already knew me ❤️. But the main challenge is how to keep my average Dopamine level high (without Dopamine, it is hard to wake up in the morning, believe me).

One of the key factors of my success and peace is, autonomy. I own my time, I own my projects, and I don't fear to cut my own path across the jungle. I'm working for a company who are more interested in the product I could deliver than how many times I spent in front of my computer. Designing under the shower is one of the most time efficient activities. When you read this, you should imagine I'm working for a small startup with a few people. And you are right, most of the time, but this is not always the case. I spend many years in the enterprise world too, and I did the same (Cloudera, IBM, Akamai just to naming some). But how?

I have found my interest, which is not technology related. My Dopamine source is community driven development. I like to work with any folks, who has the same interest nor problems. So I'm working on open source software products in the space of my team and collaborating on them with awesome internal or external people.

I think this is the easiest part of the story. I can say this freedom is just the result of something else called proactivity. When I finish a task or project, I stay to sit for a while and I imagine where we should go further based on my company's core values, kind of the project, or any customer preferences. On this way, usually I found the next step earlier than others (this is not about them competence, simply I'm the one who reached the line first). When I deliver the product, I usually suggest where to go next, how to improve things, what should be the bottleneck, etc. Most of the time they like it, and they give the responsibility && control back to me.

When I wake up in each morning, I know, today I would work ...

on something selected by me based on my interest.
on something dreamed and owned by me.
on project available for anyone.
with people who would like to achieve the same goal with me.
when my brain is in focus mode and not 9to5.

Work is not a "work" anymore. I hate to work at all ;) What I do is spending my time on the things I love the most (not counting spiritual experiences). This attitude helps me to wake up in the morning and deliver a good project. Good projects build trust in my capabilities, and lead me to the situation where my employer's best business is to let me do what I want to.

Makes sense?