Forem: suin

Kubernetes: Should You Name Your Controller "foo-bar" or "foobar"? A Survey of 13 Open-Source Projects

suin — Tue, 17 Feb 2026 23:56:55 +0000

In this post, I'll share the results of surveying the source code of 13 major open-source projects to answer a question that often comes up when building a Kubernetes Operator: how should you name controllers for multi-word resource types?

When you have a CRD (Custom Resource Definition) type like CertificateRequest or MachineDeployment, should the controller be called foo-bar-controller (hyphenated) or foobar-controller (concatenated lowercase)? There's no official guidance on this. So I dug into real-world, widely used OSS implementations to find out what the de facto standard actually looks like.

What you'll learn from this post

How multi-word CRD types are actually named across the ecosystem
The conventions 13 OSS projects use for controller names, finalizers, field managers, and more
Cross-project trend analysis revealing the most common patterns
The convention adopted by a Kubernetes SIG project — the closest thing to an "official" standard

Methodology and scope

For this survey, I cloned the repositories of 13 Kubernetes-related OSS projects and examined their naming conventions at the source code level. These findings are based on actual code, not documentation.

The projects surveyed are listed below.

I looked at six dimensions for each project.

Go file and directory names
Controller name strings
Finalizer names (the mechanism that controls cleanup before a resource is deleted)
Field manager names (identifiers for field ownership in Server-Side Apply)
Logger context values
Reconciler struct names

Master comparison table

Let's start with a side-by-side comparison of all 13 projects.

Project	Resource Type	Go File / Dir Name	Controller Name	Finalizer	Field Manager	Logger Context	Reconciler Struct
cert-manager	`CertificateRequest`	`certificaterequests/`	`"certificaterequests-issuer-acme"`	`"finalizer.acme.cert-manager.io"`	`"cert-manager-<component>"`	`"certificaterequests-issuer-acme"`	`Controller`
Argo CD	`ApplicationSet`	`applicationset_controller.go`	`"argocd-application-controller"`	`"resources-finalizer.argocd.argoproj.io"`	Generic (engine-level)	`"applicationset"`	`ApplicationSetReconciler`
Istio	`VirtualService`	Custom framework	`"istio.io/gateway-controller"`	N/A	`"pilot-discovery"`	`"KubernetesGateway"`	`autoServiceExportController`
Crossplane	`CompositeResourceDefinition`	`definition/`	`"defined/compositeresourcedefinition..."`	`"defined.apiextensions.crossplane.io"`	`"apiextensions.crossplane.io/composite"`	Same as controller name	`Reconciler`
Knative	`DomainMapping`	`domainmapping/`	`"domainmapping-controller"`	`"domainmappings.serving.knative.dev"`	Configurable (no default)	`"serving.knative.dev.DomainMapping"`	`Reconciler`
Strimzi	`KafkaConnect`	`KafkaConnectAssemblyOperator.java`	`"KafkaConnect"`	N/A (uses owner refs)	N/A (Java)	`"KafkaConnect(ns/name)"`	`KafkaConnectAssemblyOperator`
Prometheus Op	`PrometheusAgent`	`pkg/prometheus/agent/operator.go`	`"prometheusagent-controller"`	`"monitoring.coreos.com/status-cleanup"`	`"PrometheusOperator"`	`"prometheusagent-controller"`	`Operator`
Velero	`BackupStorageLocation`	`backup_storage_location_controller.go`	`"backup-storage-location"`	`"velero.io/pod-volume-finalizer"`	`"velero-cli"`	`"PodVolumeBackup"` (mixed)	`backupStorageLocationReconciler`
Tekton	`PipelineRun`	`pipelinerun/`	`"PipelineRun"`	`"chains.tekton.dev/finalizer"`	Knative defaults	`"pipelinerun-reconciler"`	`Reconciler`
Flux CD	`HelmRelease`	`helmrelease_controller.go`	`"helm-controller"`	`"finalizers.fluxcd.io"`	`"helm-controller"`	Auto (controller-runtime)	`HelmReleaseReconciler`
Cluster API	`MachineDeployment`	`machinedeployment_controller.go`	`"machinedeployment"`	`"cluster.x-k8s.io/machinedeployment"`	`"capi-machinedeployment"`	`"machinedeployment"`	`Reconciler`
KubeVirt	`VirtualMachine`	`vm/vm.go`	`"virt-controller-vm"`	`"kubevirt.io/virtualMachineControllerFinalize"`	N/A (no SSA)	N/A (custom)	`Controller`
Kyverno	`ClusterCleanupPolicy`	`cleanup/controller.go`	`"cleanup-controller"`	N/A	`"kyverno-{suffix}"`	`ControllerLogger(name)`	`controller`

Even at a glance, the concatenated lowercase pattern (foobar) stands out. Let's now look at each project in detail.

Project-by-project breakdown

1. cert-manager

cert-manager automates TLS certificate management within Kubernetes clusters. Its multi-word CRD types include CertificateRequest, ClusterIssuer, and CertificateSigningRequest.

Artifact	Convention	Examples
Go directory name	Concatenated lowercase	`certificaterequests/`, `certificatesigningrequests/`, `clusterissuers/`
Controller name	Concatenated lowercase + hyphenated suffix	`"certificaterequests-issuer-acme"`, `"certificaterequests-approver"`, `"clusterissuers"`
Finalizer	Domain-based (not resource-specific)	`"finalizer.acme.cert-manager.io"`, `"acme.cert-manager.io/finalizer"`
Field manager	Derived from UserAgent	`"cert-manager-<component>"` (e.g., `"cert-manager-test"`)
Logger context	Same as controller name	`logf.FromContext(ctx, "clusterissuers")`
Reconciler struct	`Controller` per package	Unexported `controller` or exported `Controller`

The key takeaway here is that the multi-word CRD Kind (CertificateRequest) is transformed into concatenated lowercase (certificaterequests). Hyphens only appear when separating functional suffixes like -issuer-acme or -approver. There is one exception: a directory named certificate-shim/ uses a hyphen.

2. Argo CD

Argo CD enables GitOps-based continuous delivery to Kubernetes. ApplicationSet and AppProject are its multi-word CRD types.

Artifact	Convention	Examples
Go file name	Concatenated lowercase + `_controller`	`applicationset_controller.go`, `appcontroller.go`
Controller name	Hyphenated	`"argocd-application-controller"`, `"argocd-applicationset-controller"`
Finalizer	Hyphenated + domain	`"resources-finalizer.argocd.argoproj.io"`, `"pre-delete-finalizer.argocd.argoproj.io"`
Logger context	Concatenated lowercase field key	`log.WithField("applicationset", req.NamespacedName)`
Reconciler struct	PascalCase	`ApplicationSetReconciler`, `ApplicationController`

Deployment names use hyphenation (argocd-application-controller), but Go file names and logger keys use concatenated lowercase (applicationset).

3. Istio

Istio provides a service mesh. Its CRD types include VirtualService, DestinationRule, ServiceEntry, and WorkloadEntry.

Artifact	Convention	Examples
Go file name	Concatenated lowercase	`autoserviceexportcontroller.go`, `deploymentcontroller.go`, `configcontroller.go`
Controller name	Domain + hyphenated	`"istio.io/gateway-controller"`, `"istio.io/inference-pool-controller"`, `"istio.io/mesh-controller"`
Field manager	Hyphenated	`"pilot-discovery"`, `"istio-operator"`, `"istio.io/inference-pool-controller"`
Schema plurals	Concatenated lowercase	`"virtualservices"`, `"destinationrules"`
Log collection names	PascalCase	`"KubernetesGateway"`, `"HTTPRoute"`, `"GatewayClasses"`
Reconciler struct	camelCase	`autoServiceExportController`

Istio uses its own custom controller framework. File names are concatenated lowercase, but controller identifier strings use hyphenation. Log collection names preserve PascalCase as-is, which is also notable.

4. Crossplane

Crossplane manages cloud infrastructure through Kubernetes. It has particularly long CRD type names like CompositeResourceDefinition and ManagedResourceActivationPolicy.

Artifact	Convention	Examples
Go directory name	Semantic / abbreviated	`definition/`, `activationpolicy/`, `watchoperation/`, `cronoperation/`
Controller name	`"prefix/lowercasedKind.group"`	`"defined/compositeresourcedefinition.apiextensions.crossplane.io"`, `"mrap/managedresourceactivationpolicy"`
Finalizer	Concatenated lowercase as subdomain	`"defined.apiextensions.crossplane.io"`, `"watchoperation.ops.crossplane.io"`, `"composite.apiextensions.crossplane.io"`
Field manager	Domain path	`"apiextensions.crossplane.io/composite"`, `"apiextensions.crossplane.io/managed"`
Logger context	Controller name value	`WithValues("controller", controllerName)`
Reconciler struct	`Reconciler` per package	`Reconciler`

Controller names use the lowercased Kind directly with no separators at all (compositeresourcedefinition, managedresourceactivationpolicy). Finalizers similarly use concatenated lowercase as subdomains.

5. Knative

Knative is a platform for running serverless workloads on Kubernetes. Its multi-word CRD types include DomainMapping, ServerlessService, PodAutoscaler, KnativeServing, and KnativeEventing.

Artifact	Convention	Examples
Go directory name	Concatenated lowercase	`domainmapping/`, `serverlessservice/`, `knativeserving/`, `knativeeventing/`
Controller name	Concatenated lowercase + `-controller`	`"domainmapping-controller"`, `"serverlessservice-controller"`, `"podautoscaler-controller"`, `"knativeserving-controller"`
Finalizer	Concatenated lowercase plural + API group	`"domainmappings.serving.knative.dev"`, `"serverlessservices.networking.internal.knative.dev"`, `"knativeservings.operator.knative.dev"`
Log Kind	PascalCase	`"serving.knative.dev.DomainMapping"`
Reconciler struct	`Reconciler` per package (code-generated)	`reconcilerImpl` (generated), `Reconciler` (user-implemented)

What makes Knative stand out is that its code generator enforces a consistent pattern. Agent names follow the lowercased-kind-controller format uniformly. This is the most programmatic and consistent approach among all 13 projects.

6. Strimzi Kafka Operator

Strimzi is an Operator for running Apache Kafka on Kubernetes. It is implemented in Java and has CRD types such as KafkaConnect, KafkaMirrorMaker2, KafkaBridge, KafkaRebalance, and KafkaNodePool.

Artifact	Convention	Examples
Java class name	PascalCase + `AssemblyOperator`	`KafkaConnectAssemblyOperator.java`, `KafkaMirrorMaker2AssemblyOperator.java`
Resource Kind	PascalCase	`"KafkaConnect"`, `"KafkaMirrorMaker2"`, `"KafkaBridge"`
Resource plural	Concatenated lowercase	`"kafkaconnects"`, `"kafkamirrormaker2s"`, `"kafkanodepools"`
Labels	PascalCase values	`"strimzi.io/kind": "KafkaConnect"`
Lock names	PascalCase Kind	`"lock::ns::KafkaConnect::name"`
Log markers	PascalCase	`"KafkaConnect(namespace/name)"`

Strimzi uses PascalCase (the Kind itself) in most runtime contexts. Plural forms are concatenated lowercase (kafkaconnects), following the standard Kubernetes API convention.

7. Prometheus Operator

Prometheus Operator automates Prometheus operations on Kubernetes. Its CRD types include ServiceMonitor, PodMonitor, ThanosRuler, PrometheusAgent, ScrapeConfig, and AlertmanagerConfig.

Artifact	Convention	Examples
Go directory / file	`operator.go` per package	`pkg/alertmanager/operator.go`, `pkg/prometheus/agent/operator.go`, `pkg/thanos/operator.go`
Controller name	Concatenated lowercase + `-controller`	`"prometheusagent-controller"`, `"alertmanager-controller"`, `"thanos-controller"`
Finalizer	Shared (not type-specific)	`"monitoring.coreos.com/status-cleanup"`
Field manager	Shared PascalCase	`"PrometheusOperator"`
managed-by label	Hyphenated	`"prometheus-operator"`
Logger context	Controller name value	`"component": "prometheusagent-controller"`
Reconciler struct	`Operator` per package	`alertmanager.Operator`, `thanos.Operator`

What's notable here is how multi-word CRD types are concatenated. PrometheusAgent becomes "prometheusagent-controller" — not "prometheus-agent-controller". Additionally, ThanosRuler is shortened to just "thanos-controller".

8. Velero

Velero handles backup and restore for Kubernetes clusters. Its CRD types include BackupStorageLocation, VolumeSnapshotLocation, ServerStatusRequest, PodVolumeBackup, PodVolumeRestore, DataDownload, DataUpload, and DownloadRequest.

Artifact	Convention	Examples
Go file name	Snake case	`backup_storage_location_controller.go`, `pod_volume_backup_controller.go`, `server_status_request_controller.go`
Controller name	Hyphenated (kebab-case)	`"backup-storage-location"`, `"pod-volume-backup"`, `"server-status-request"`, `"data-download"`
Finalizer	Hyphenated + domain	`"velero.io/pod-volume-finalizer"`, `"velero.io/data-upload-download-finalizer"`
Field manager	Per-binary	`"velero-cli"`
Logger context	Mixed	`"controller": "PodVolumeBackup"` (PascalCase) in constructors vs. `"controller": "podvolumebackup"` (concatenated) in Reconcile
Reconciler struct	Mixed export	`backupStorageLocationReconciler` (unexported), `PodVolumeBackupReconciler` (exported)

Velero is the only project among all 13 that consistently uses hyphenated word splitting for controller names. Each word from the PascalCase type is separated by a hyphen. Its use of snake case for Go file names is another unique trait not seen in any other project.

9. Tekton Pipelines

Tekton is a framework for building CI/CD pipelines on Kubernetes. Its CRD types include PipelineRun, TaskRun, CustomRun, and ResolutionRequest.

Artifact	Convention	Examples
Go directory name	Concatenated lowercase	`pipelinerun/`, `taskrun/`, `customrun/`, `resolutionrequest/`
Controller name (AgentName)	PascalCase (Kind as-is)	`"PipelineRun"`, `"TaskRun"`, `"CustomRun"`
Finalizer	Domain-based	`"chains.tekton.dev/finalizer"` (from Tekton Chains)
managed-by	Domain path	`"tekton.dev/pipeline"`
Tracer name	Concatenated lowercase + `-reconciler`	`"pipelinerun-reconciler"`, `"taskrun-reconciler"`
Reconciler struct	`Reconciler` per package	`Reconciler` (distinguished by package)

Tekton takes the unique approach of using the PascalCase Kind directly as the controller agent name: "PipelineRun", not "pipelinerun" or "pipeline-run". Directory names and tracer names, on the other hand, use concatenated lowercase.

10. Flux CD

Flux CD is a GitOps toolkit. Its CRD types include GitRepository, HelmRelease, HelmChart, OCIRepository, HelmRepository, and ImageUpdateAutomation.

Artifact	Convention	Examples
Go file name	Concatenated lowercase + `_controller.go`	`helmrelease_controller.go`, `gitrepository_controller.go`, `ocirepository_controller.go`, `imageupdateautomation_controller.go`
Controller name	Per-binary (not per-CRD)	`"source-controller"`, `"helm-controller"`, `"image-automation-controller"`
Finalizer	Shared across all types	`"finalizers.fluxcd.io"`
Field manager	Binary name	`"source-controller"`, `"helm-controller"`
Reconciler struct	PascalCase + Reconciler	`HelmReleaseReconciler`, `GitRepositoryReconciler`, `OCIRepositoryReconciler`

Flux doesn't assign per-CRD controller names at all. The controller name is simply the binary name (the executable itself). File names use concatenated lowercase.

11. Cluster API (CAPI)

Cluster API is a Kubernetes SIG project for managing cluster lifecycles. It has a large number of multi-word CRD types: MachineDeployment, MachineSet, MachineHealthCheck, ClusterClass, ClusterResourceSet, ClusterResourceSetBinding, MachinePool, and more.

Artifact	Convention	Examples
Go file name	Concatenated lowercase + `_controller.go`	`machinedeployment_controller.go`, `machinehealthcheck_controller.go`, `clusterresourceset_controller.go`
Go directory name	Concatenated lowercase	`machinedeployment/`, `machinehealthcheck/`, `clusterresourceset/`
Controller name	Concatenated lowercase	`"machinedeployment"`, `"machinehealthcheck"`, `"clusterresourceset"`, `"clusterresourcesetbinding"`
Event recorder	Concatenated lowercase + `-controller`	`"machinedeployment-controller"`, `"machinehealthcheck-controller"`
Finalizer	Concatenated lowercase in domain	`"cluster.x-k8s.io/machinedeployment"`, `"machinedeployment.topology.cluster.x-k8s.io"`, `"machinepool.cluster.x-k8s.io"`
Field manager (SSA)	`capi-` + concatenated lowercase	`"capi-machinedeployment"`, `"capi-machineset"`
Logger context	Concatenated lowercase	`WithValues("controller", "machinedeployment")`, `WithValues("controller", "clusterresourcesetbinding")`
SSA cache	Concatenated lowercase	`ssa.NewCache("machinedeployment")`
Reconciler struct	`Reconciler` per package	`machinedeployment.Reconciler`, `machinehealthcheck.Reconciler`

Cluster API has the most comprehensive and consistent naming convention of all 13 projects. Every single artifact uses concatenated lowercase for the resource type. MachineDeployment becomes machinedeployment everywhere, without exception.

As a Kubernetes SIG project, this convention can be considered the closest thing to an "official" standard.

12. KubeVirt

KubeVirt enables running virtual machines on Kubernetes. It has particularly long CRD type names: VirtualMachine, VirtualMachineInstance, VirtualMachineInstanceReplicaSet, VirtualMachinePool, and VirtualMachineInstanceMigration.

Artifact	Convention	Examples
Go directory / file	Abbreviated	`vm/vm.go`, `vmi/vmi.go`, `replicaset/replicaset.go`, `pool/pool.go`, `migration/migration.go`
Work queue name	Abbreviated + hyphenated prefix	`"virt-controller-vm"`, `"virt-controller-vmi"`, `"virt-controller-replicaset"`
Event recorder	Concatenated lowercase + `-controller`	`"virtualmachine-controller"`, `"virtualmachinereplicaset-controller"`, `"virtualmachinepool-controller"`
Finalizer	camelCase	`"kubevirt.io/foregroundDeleteVirtualMachine"`, `"kubevirt.io/virtualMachineControllerFinalize"`, `"kubevirt.io/migrationJobFinalize"`
Controller struct	`Controller` per package	`vm.Controller`, `vmi.Controller`, `migration.Controller`

KubeVirt uses abbreviations (VM, VMI) aggressively in its internal code, while event recorders use concatenated lowercase (virtualmachine). Its use of camelCase strings for finalizers is unique among all 13 projects.

13. Kyverno

Kyverno is a policy engine for Kubernetes. Its CRD types include ClusterPolicy, ClusterCleanupPolicy, PolicyReport, and GlobalContextEntry.

Artifact	Convention	Examples
Go directory name	By functional concern	`cleanup/`, `policycache/`, `policystatus/`, `globalcontext/`
Controller name	Hyphenated (concern-based)	`"cleanup-controller"`, `"policycache-controller"`, `"global-context"`, `"admissionpolicy-generator"`
Field manager	Prefixed	`"kyverno-{suffix}"`
Logger context	Controller name	`logging.ControllerLogger(ControllerName)`
Reconciler struct	Unexported generic name	`controller` (per package)

What distinguishes Kyverno is that controllers are named by functional concern rather than CRD type. When multi-word concepts appear, it mixes concatenated lowercase (policycache, admissionpolicy) with hyphenation (global-context), resulting in some inconsistency.

Cross-project analysis

Based on the individual findings above, let's now analyze overall trends by artifact type.

Go file and directory names

Pattern	Projects Using It	Share
Concatenated lowercase (`foobar_controller.go`, `foobar/`)	cert-manager, Argo CD, Knative, Tekton, Flux, Cluster API, Crossplane, Prometheus Op	~70%
Snake case (`foo_bar_controller.go`)	Velero	~8%
Abbreviated (`vm.go`)	KubeVirt	~8%
By concern (`controller.go` in a semantic directory)	Kyverno, Crossplane (partially)	~15%

Concatenated lowercase is the overwhelming favorite. FooBar becomes foobar_controller.go or foobar/controller.go.

Controller name strings

Pattern	Projects Using It	Examples
Concatenated lowercase (bare or + `-controller`)	Cluster API, Knative, Prometheus Op, cert-manager, Crossplane	`"machinedeployment"`, `"domainmapping-controller"`, `"prometheusagent-controller"`
Hyphenated (word-split)	Velero	`"backup-storage-location"`, `"pod-volume-backup"`
PascalCase (Kind as-is)	Tekton, Strimzi, Flux (logger only)	`"PipelineRun"`, `"KafkaConnect"`
Binary name (not per-CRD)	Flux, Istio	`"helm-controller"`, `"pilot-discovery"`
By concern	Kyverno	`"cleanup-controller"`

Again, concatenated lowercase is the most common. FooBar becomes "foobar" or "foobar-controller".

Finalizer names

Pattern	Projects Using It	Examples
Concatenated lowercase in domain	Cluster API, Knative, Crossplane	`"cluster.x-k8s.io/machinedeployment"`, `"domainmappings.serving.knative.dev"`
Shared / generic	Flux, Prometheus Op, cert-manager	`"finalizers.fluxcd.io"`, `"monitoring.coreos.com/status-cleanup"`
Hyphenated	Velero	`"velero.io/pod-volume-finalizer"`
camelCase	KubeVirt	`"kubevirt.io/virtualMachineControllerFinalize"`

Field manager names

Pattern	Projects Using It	Examples
Prefix + concatenated lowercase	Cluster API	`"capi-machinedeployment"`
Binary name	Flux, Istio	`"source-controller"`, `"pilot-discovery"`
Shared operator name	Prometheus Op	`"PrometheusOperator"`
Domain path	Crossplane	`"apiextensions.crossplane.io/composite"`

Reconciler struct names

Pattern	Projects Using It	Share
Generic `Reconciler` per package	Cluster API, Knative, Tekton, Crossplane	~40%
PascalCase `FooBarReconciler`	Flux, Argo CD	~20%
`Operator` per package	Prometheus Op	~8%
`Controller` per package	KubeVirt, cert-manager	~15%
Unexported `controller`	Kyverno	~8%

Conclusions

The dominant convention: concatenated lowercase

The clear winner across all naming dimensions is concatenated lowercase — lowercasing the PascalCase Kind with no separator.

FooBar → foobar

Here's a breakdown of adoption rates for this pattern.

~70% of projects use it for file/directory names
~50% of projects use it for controller name strings
~40% of projects use it for finalizer domain components
100% of projects use it for Kubernetes API resource plurals (this is a Kubernetes API standard itself)

Who uses hyphenation?

Only Velero consistently word-splits CRD types with hyphens ("backup-storage-location", "pod-volume-backup") — that's 1 out of 13 projects.

Kyverno partially uses hyphens ("global-context"), but also mixes in concatenated lowercase (policycache, admissionpolicy), so it is not consistent.

The `-controller` suffix

When appending a -controller suffix to a name, every single project connects it with a hyphen. It's always "foobar-controller", never "foobarcontroller". This is universal.

Recommended naming convention

Based on the survey results, here is a recommended set of conventions. The Cluster API conventions (a Kubernetes SIG project) are particularly worth following, as they are the most comprehensive and consistent of any project surveyed.

Artifact	Recommended Convention	Example for `FooBar`
Go file name	Concatenated lowercase + underscore + `controller.go`	`foobar_controller.go`
Go directory	Concatenated lowercase	`foobar/`
Controller name	Concatenated lowercase	`"foobar"`
Controller name (with suffix)	Concatenated lowercase + hyphen + `controller`	`"foobar-controller"`
Finalizer	Concatenated lowercase in domain	`"example.io/foobar"` or `"foobar.example.io"`
Field manager	Prefix + concatenated lowercase	`"myoperator-foobar"`
Logger context	Concatenated lowercase	`"foobar"`
Reconciler struct	PascalCase or generic per package	`FooBarReconciler` or `Reconciler`

Why concatenated lowercase wins

There are several reasons why concatenated lowercase is so widely adopted.

First, it aligns with the Kubernetes API's own conventions. The standard resource plurals are already concatenated lowercase: deployments, replicasets, statefulsets, daemonsets. CRD resources naturally follow this same pattern.

Second, it fits well with Go package naming conventions. Go packages are conventionally single lowercase words, and a name like machinedeployment fits right in.

Third, it eliminates ambiguity. With hyphenation, splitting FooBar into foo-bar is straightforward, but FooBarBaz could plausibly be foo-bar-baz or foo-barbaz — the word boundaries aren't always obvious. With concatenated lowercase, you simply lowercase the Kind as-is. It's a mechanical transformation with no room for interpretation.

And perhaps the most compelling reason: Cluster API, a Kubernetes SIG project, uses this convention with complete consistency. If you want to follow the crowd in the Kubernetes ecosystem, this is the convention to pick.

Hyphenation has its merits too

That said, hyphenation has legitimate advantages.

There's no denying that backup-storage-location is easier for humans to read than backupstoragelocation. In contexts where humans read names directly — deployment names, CLI output, log messages — readability is a significant benefit.

Additionally, hyphenated names are the standard convention in Kubernetes label values, annotation keys, and resource names in manifests.

In other words, it comes down to a choice: concatenated lowercase for consistency with APIs and code, or hyphenation for human readability. Neither is wrong — but the ecosystem has spoken, and concatenated lowercase is the overwhelming favorite.

Final thoughts

In this post, I surveyed the source code of 13 Kubernetes-related OSS projects and mapped out the real-world naming conventions for multi-word CRD types. The verdict: concatenated lowercase (foobar) is the de facto standard, with Velero being the only project that consistently uses hyphenation (foo-bar).

If you're building a Kubernetes Operator and unsure which convention to follow, Cluster API's approach is a safe bet. That said, there's no absolute right answer here. The most pragmatic approach is to prioritize consistency within your own project and go with whatever your team finds readable.

Thanks for reading to the end! I tweet about tech topics that don't make it into my blog posts, so feel free to follow me if you're interested → Twitter@suin

controller-runtime: What Happens When You Do Partial Server-Side Apply?

suin — Thu, 18 Sep 2025 00:35:38 +0000

In this post, I'll explore what happens when you repeatedly apply partial manifests using Kubernetes Server-Side Apply (SSA). Specifically, I'll investigate through experiments what occurs when you omit fields that were previously managed by a field manager in subsequent applies.

While developing controllers, I found myself wondering: "What happens to fields I don't include when applying partial manifests with SSA?" If only the submitted portions are merged as a diff, we could simplify our code by sending only the fields we want to manage. On the other hand, if we need to send all managed fields every time, we need to be conscious of this when coding, or we might accidentally create bugs that unintentionally delete fields.

Developing controllers with incorrect assumptions could lead to bugs that unintentionally destroy resources. While this is about understanding the basic specifications of SSA, I felt it was essential to have a clear understanding, which is why I'm writing this post.

The experimental code referenced in this article is available in the following repository:

https://github.com/suinplayground/controller-runtime/tree/main/02-server-side-apply-partials

What You'll Learn

The behavior when applying partial manifests with Server-Side Apply
Field ownership and the role of field managers
The deletion mechanism that occurs when managed fields are omitted
Collaborative management by multiple field managers

Purpose of the Experiment

The goal of this experiment is to understand what happens when a field manager omits a field in a subsequent apply that was included in a previously applied manifest.

In real controller development, you might encounter scenarios like: "Initially I was managing the breed field, but later I only want to update the color field." What happens to the breed field in this case?

Experiment 1: What Happens When You Omit Fields Managed by the Same Manager?

In this experiment, I'll use a custom resource called Cat. I'll specify cat-owner as the "field manager" that manages field ownership.

Step 1: First Apply (breed field only)

First, I'll apply a partial manifest containing only spec.breed to the Cat resource my-cat.

cat := applycatv1.Cat("my-cat", "default").
    WithSpec(applycatv1.CatSpec().
        WithBreed("Maine Coon"))

err := cl.Apply(ctx, cat, client.FieldOwner("cat-owner"), client.ForceOwnership)

After this operation, the resource state looks like this. You can see that cat-owner managing spec.breed is recorded in managedFields.

"spec": {
  "breed": "Maine Coon"
},
"managedFields": [
  {
    "manager": "cat-owner",
    ...
    "fieldsV1": {
      "f:spec": {
        "f:breed": {}
      }
    }
  }
]

Step 2: Second Apply (color field only, breed omitted)

Next, I'll apply a manifest containing only spec.color to the same my-cat resource, using the same cat-owner. The crucial point is that this manifest doesn't include spec.breed.

cat := applycatv1.Cat("my-cat", "default").
    WithSpec(applycatv1.CatSpec().
        WithColor("Black")) // no breed

err := cl.Apply(ctx, cat, client.FieldOwner("cat-owner"), client.ForceOwnership)

Observation

After the second Apply, while spec.color was added to the resource, the spec.breed field completely disappeared.

"spec": {
  "color": "Black"
},
"managedFields": [
  {
    "manager": "cat-owner",
    ...
    "fieldsV1": {
      "f:spec": {
        "f:color": {}
      }
    }
  }
]

Looking at managedFields, we can see that cat-owner's managed target has been updated from spec.breed to spec.color.

Why Was the Field Deleted?

This behavior relates to Server-Side Apply's "declarative ownership management." The Kubernetes official documentation explains exactly this case:

If you remove a field from a manifest and apply that manifest, Server-Side Apply checks if there are any other field managers that also own the field. If the field is not owned by any other field managers, it is either deleted from the live object or reset to its default value, if it has one. The same rule applies to associative list or map items.

-- Field management - Kubernetes

This explanation perfectly describes our experimental results:

In the first Apply, cat-owner became the sole owner of the spec.breed field
In the second Apply, cat-owner didn't include spec.breed in the manifest. This is interpreted as "I no longer want to manage spec.breed"
Since there were no other owners of spec.breed, Server-Side Apply deleted this field from the live object

This is a cleanup feature that prevents unmanaged fields from persisting indefinitely. To avoid unintentionally deleting fields, you need to understand that you must include all fields you want to manage in every Apply request.

Supplementary Experiment: What Happens When Another Field Manager Appears?

So what happens when another field manager enters the picture? Let's look at the ownership aspect of SSA.

Step 3: Third Apply by a Different Manager

From the previous state (where only spec.color exists), a new field manager called age-controller applies a manifest containing only spec.age.

Observation

After applying, the resource has both spec.color and spec.age. The spec.color managed by cat-owner remains unaffected.

"spec": {
  "color": "Black",
  "age": 3
},
"managedFields": [
  {
    "manager": "age-controller",
    ...
    "fieldsV1": {
      "f:spec": {
        "f:age": {}
      }
    }
  },
  {
    "manager": "cat-owner",
    ...
    "fieldsV1": {
      "f:spec": {
        "f:color": {}
      }
    }
  }
]

managedFields records two entries for age-controller and cat-owner, clearly tracking which fields each owns.

age-controller isn't omitting any fields it previously owned (none existed in this case), so it simply claimed ownership of spec.age. It didn't interfere with fields owned by cat-owner.

Summary

From this experiment, we can understand two key aspects of using Server-Side Apply:

Manifest Completeness = Include All Fields: When a field manager omits a field it was managing from the next Apply manifest, that field will be deleted (if no other owners exist). Therefore, always include fields under management in your manifest to maintain manifest completeness.
Completeness is Per-Owner: Multiple field managers can safely manage different fields of the same resource without interfering with each other, so you don't need to include fields managed by others in your manifest.

Server-Side Apply is a crucial feature in Kubernetes environments where multiple controllers need to collaboratively manage a single resource. The phenomenon of fields "disappearing" isn't a bug—it's a powerful "feature" that makes declarative configuration management more robust. Understanding this behavior enables safer resource management.

Thank you for reading to the end!

Flagger Does Not Manage Services with Names Different from Their Deployments

suin — Fri, 18 Apr 2025 02:37:35 +0000

Flagger Does Not Manage Services with Names Different from Their Deployments

Typically, Flagger creates Services with the same name as their Deployments. If a Service with the same name already exists, Flagger is known to take it under management (making it a reconciliation target) and modify properties like spec.selector.app.

But what happens when a Service is related to a Deployment but has a different name? Will Flagger recognize and manage it, or will it leave it alone? We conducted an investigation to find out.

Test Environment

Kubernetes: Kind v1.32.2
Flagger: v1.41.0
Gateway API: v1.2.0
Envoy Gateway: v1.3.2

Testing Method

Set up the test environment
- Create a Kind cluster
- Install Gateway API (v1.2.0)
- Install Cert Manager (v1.17.1)
- Install Envoy Gateway (v1.3.2)
- Create GatewayClass and Gateway
- Install Flagger (v1.41.0)

Detailed setup procedure

Create Kind cluster
```
kind create cluster
```

Install Gateway API (v1.2.0)

kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.0/standard-install.yaml

Install Cert Manager (v1.17.1)

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.17.1/cert-manager.yaml
kubectl wait --for=condition=available --timeout=60s deployment/cert-manager -n cert-manager
kubectl wait --for=condition=available --timeout=60s deployment/cert-manager-webhook -n cert-manager
kubectl wait --for=condition=available --timeout=60s deployment/cert-manager-cainjector -n cert-manager

Install Envoy Gateway (v1.3.2)

kubectl apply --server-side -f https://github.com/envoyproxy/gateway/releases/download/v1.3.2/install.yaml
kubectl wait --for=condition=available --timeout=600s deployment/envoy-gateway -n envoy-gateway-system

Create GatewayClass and Gateway

kubectl apply -f manifests/gatewayclass.yaml
kubectl wait --for=condition=accepted --timeout=60s gatewayclass/eg
kubectl apply -f manifests/gateway.yaml
kubectl wait --for=condition=programmed --timeout=60s gateway/eg

gatewayclass.yaml

apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
name: eg
spec:
controllerName: gateway.envoyproxy.io/gatewayclass-controller
parametersRef:
    group: gateway.envoyproxy.io
    kind: EnvoyProxy
    name: clusterip-config
    namespace: envoy-gateway-system
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
name: clusterip-config
namespace: envoy-gateway-system
spec:
provider:
    type: Kubernetes
    kubernetes:
    envoyService:
        type: ClusterIP

gateway.yaml

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: eg
spec:
gatewayClassName: eg
listeners:
    - name: http
    protocol: HTTP
    port: 80

Install Flagger (v1.41.0)

kubectl apply -f https://raw.githubusercontent.com/fluxcd/flagger/v1.41.0/artifacts/flagger/crd.yaml
kubectl get ns flagger-system || kubectl create ns flagger-system
helm repo add flagger https://flagger.app
helm upgrade -i flagger flagger/flagger \
--version 1.41.0 \
--namespace flagger-system \
--set prometheus.install=false \
--set meshProvider=gatewayapi:v1 \
--set metricsServer=none

Deploy the following resources:
- Deployment name: podinfo
- Service name: podinfo-different-name (different from Deployment name)
- Canary name: podinfo (same as Deployment name)

   kubectl apply -f manifests/deployment.yaml
   kubectl apply -f manifests/service.yaml
   kubectl apply -f manifests/canary.yaml

Content of the deployed manifest files

### deployment.yaml

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: podinfo
   spec:
     replicas: 2
     selector:
       matchLabels:
         app: podinfo
     template:
       metadata:
         labels:
           app: podinfo
       spec:
         containers:
         - name: podinfo
           image: stefanprodan/podinfo:latest
           ports:
           - containerPort: 9898

### service.yaml

   apiVersion: v1
   kind: Service
   metadata:
     name: podinfo-different-name
   spec:
     selector:
       app: podinfo
     ports:
     - port: 9898
       targetPort: 9898

### canary.yaml

   apiVersion: flagger.app/v1beta1
   kind: Canary
   metadata:
     name: podinfo
   spec:
     targetRef:
       apiVersion: apps/v1
       kind: Deployment
       name: podinfo
     progressDeadlineSeconds: 60
     service:
       port: 9898
       targetPort: 9898
       gatewayRefs:
         - name: eg
           namespace: default
     analysis:
       interval: 1s
       threshold: 5
       maxWeight: 50
       stepWeight: 10

Test Results

Services after deployment

$ kubectl get services
NAME                     TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
kubernetes               ClusterIP   10.96.0.1       <none>        443/TCP    8m30s
podinfo                  ClusterIP   10.96.126.169   <none>        9898/TCP   5m39s
podinfo-canary           ClusterIP   10.96.20.177    <none>        9898/TCP   5m49s
podinfo-different-name   ClusterIP   10.96.141.217   <none>        9898/TCP   6m12s
podinfo-primary          ClusterIP   10.96.130.137   <none>        9898/TCP   5m49s

Note: The podinfo service (same name as Deployment) is created slightly after the Canary resource is created.

Original Service (podinfo-different-name)

apiVersion: v1
kind: Service
metadata:
  # No ownerReferences - not managed by Flagger
  name: podinfo-different-name
  namespace: default
spec:
  # Selector remains unchanged
  selector:
    app: podinfo
  ports:
  - port: 9898
    targetPort: 9898

podinfo-primary service created by Flagger

apiVersion: v1
kind: Service
metadata:
  labels:
    app: podinfo-primary
  name: podinfo-primary
  namespace: default
  ownerReferences:
  - apiVersion: flagger.app/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: Canary
    name: podinfo
    uid: c388a7b6-6dc9-4aa9-b9f1-e55ad8f31575
spec:
  selector:
    app: podinfo-primary
  ports:
  - name: http
    port: 9898
    targetPort: 9898

podinfo-canary service created by Flagger

apiVersion: v1
kind: Service
metadata:
  labels:
    app: podinfo-canary
  name: podinfo-canary
  namespace: default
  ownerReferences:
  - apiVersion: flagger.app/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: Canary
    name: podinfo
    uid: c388a7b6-6dc9-4aa9-b9f1-e55ad8f31575
spec:
  selector:
    app: podinfo
  ports:
  - name: http
    port: 9898
    targetPort: 9898

Service with the same name as Deployment (podinfo) created by Flagger

apiVersion: v1
kind: Service
metadata:
  annotations:
    helm.toolkit.fluxcd.io/driftDetection: disabled
    kustomize.toolkit.fluxcd.io/reconcile: disabled
  labels:
    app: podinfo
  name: podinfo
  namespace: default
  ownerReferences:
  - apiVersion: flagger.app/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: Canary
    name: podinfo
    uid: c388a7b6-6dc9-4aa9-b9f1-e55ad8f31575
spec:
  selector:
    app: podinfo-primary  # Routes traffic to primary
  ports:
  - name: http
    port: 9898
    targetPort: 9898

Conclusion

Based on our investigation, we confirmed the following:

Flagger does not recognize or manage Services with names different from their Deployments
- The Service podinfo-different-name, which has a different name from Deployment podinfo, was not controlled or modified by Flagger
Flagger creates its own services
- When deploying a Canary resource, Flagger created the following three Services:
  - podinfo-primary - Service that routes traffic to primary version pods
  - podinfo-canary - Service that routes traffic to canary version pods
  - podinfo - Service with the same name as the Deployment (created slightly later). This receives actual end-user traffic and internally routes to podinfo-primary
- All of these Services are managed by Flagger and have ownerReferences set
The original Service remains unchanged
- The podinfo-different-name Service remains in its original state without being modified by Flagger
- Its selector and other settings remain unchanged
Flagger's internal mechanism
- Flagger creates a service with the same name as the Deployment (podinfo) to receive end-user traffic and internally controls traffic between podinfo-primary and podinfo-canary
- This control enables canary deployments that gradually shift traffic to the canary version

From this investigation, it is clear that Flagger does not recognize or manage services with names that differ from their deployments. This confirms that creating a Service with a name different from its Deployment is an effective strategy to avoid automatic management by Flagger.

The 'hosts' Field in Flagger Canary Resources Can Be Omitted

suin — Tue, 15 Apr 2025 07:44:17 +0000

TL;DR

In Flagger's Canary resources, the hosts field can be safely omitted without affecting functionality.

Even when integrating with the Gateway API, you can simply configure gatewayRefs without specifying the hosts field, and HTTPRoutes will be generated correctly, allowing canary releases to function properly.

Background and Purpose

When using Flagger for canary releases with Gateway API integration, you typically specify the hosts field as follows:

spec:
  service:
    port: 9898
    targetPort: 9898
    hosts:
      - podinfo.example.com
    gatewayRefs:
      - name: gateway-name
        namespace: gateway-namespace

However, it wasn't clear whether the hosts field is mandatory or what happens when it's omitted. This article investigates the behavior when the hosts field is omitted.

Environment

The following environment was used for testing:

Kubernetes: Kind cluster
Gateway API: Envoy Gateway
Flagger: Latest version

Testing Procedure

The following steps can be reproduced using kubectl commands.

1. Create a Gateway Resource

First, create a basic Gateway resource:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: eg
spec:
  gatewayClassName: eg
  listeners:
    - name: http
      protocol: HTTP
      port: 80

kubectl apply -f gateway.yaml

2. Create the Initial Deployment

Deploy a test application (podinfo):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: podinfo
  labels:
    app: podinfo
spec:
  replicas: 2
  selector:
    matchLabels:
      app: podinfo
  template:
    metadata:
      labels:
        app: podinfo
    spec:
      containers:
      - name: podinfo
        image: stefanprodan/podinfo:6.0.0
        imagePullPolicy: IfNotPresent
        ports:
        - name: http
          containerPort: 9898
          protocol: TCP
        resources:
          limits:
            cpu: 500m
            memory: 256Mi
          requests:
            cpu: 100m
            memory: 64Mi
        livenessProbe:
          httpGet:
            path: /healthz
            port: 9898
          initialDelaySeconds: 5
          timeoutSeconds: 5
        readinessProbe:
          httpGet:
            path: /readyz
            port: 9898
          initialDelaySeconds: 5
          timeoutSeconds: 5

kubectl apply -f deployment.yaml

3. Create a Canary Resource Without the hosts Field

This is the core of our test. Create a Canary resource without the hosts field:

apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  name: podinfo
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: podinfo
  progressDeadlineSeconds: 60
  service:
    port: 9898
    targetPort: 9898
    # hosts field is omitted
    gatewayRefs:
      - name: eg
        namespace: default  # Change to match your actual namespace
  analysis:
    interval: 1s
    threshold: 5
    maxWeight: 50
    stepWeight: 10

kubectl apply -f canary.yaml

4. Check the Canary Resource Status

Verify that the Canary resource has been initialized correctly:

kubectl get canary podinfo

Example output:

NAME      STATUS        WEIGHT   LASTTRANSITIONTIME
podinfo   Initialized   0        2025-04-15T07:05:52Z

Also, check that the HTTPRoute has been generated correctly:

kubectl get httproute podinfo

Example output:

NAME      HOSTNAMES   AGE
podinfo               2m

Note that the HOSTNAMES field is empty, indicating that no specific hostnames are set because we omitted the hosts field.

5. Verify HTTP Requests

Identify the service created by Envoy Gateway and send a request to it:

# Get the service name
SERVICE_NAME=$(kubectl get service -n envoy-gateway-system -l gateway.envoyproxy.io/owning-gateway-namespace=default -o jsonpath='{.items[0].metadata.name}')
echo "Service name: ${SERVICE_NAME}"

# Send an HTTP request using curl
kubectl run curl-test --image=curlimages/curl --restart=Never -i --rm -- \
  curl -v http://${SERVICE_NAME}.envoy-gateway-system.svc.cluster.local

Example output (excerpt):

Service name: envoy-default-eg-xxxxxxxx
{
  "hostname": "podinfo-primary-xxxxxxxxxx-xxxxx",
  "version": "6.0.0",
  "message": "greetings from podinfo v6.0.0",
  ...
}

This confirms that we can access the application through the Gateway even without specifying the hosts field.

6. Update the Version

Next, update the Deployment to deploy a new version:

kubectl patch deployment podinfo -p '{"spec":{"template":{"spec":{"containers":[{"name":"podinfo","image":"stefanprodan/podinfo:6.1.0"}]}}}}'

7. Monitor the Canary Release Progress

Check that Flagger is progressing with the canary release:

kubectl get canary podinfo -w

Example output:

NAME      STATUS      WEIGHT   LASTTRANSITIONTIME
podinfo   Progressing 0        2025-04-15T07:06:10Z
podinfo   Progressing 10       2025-04-15T07:06:20Z
podinfo   Progressing 20       2025-04-15T07:06:30Z
...
podinfo   Promoting   50       2025-04-15T07:07:00Z
...
podinfo   Succeeded   0        2025-04-15T07:07:20Z

8. Verify HTTP Requests After Update

Finally, confirm that we can access the updated version:

# Get the service name
SERVICE_NAME=$(kubectl get service -n envoy-gateway-system -l gateway.envoyproxy.io/owning-gateway-namespace=default -o jsonpath='{.items[0].metadata.name}')

# Send an HTTP request using curl
kubectl run curl-test-after-update --image=curlimages/curl --restart=Never -i --rm -- \
  curl -v http://${SERVICE_NAME}.envoy-gateway-system.svc.cluster.local

Example output (excerpt):

{
  "hostname": "podinfo-primary-xxxxxxxxxx-xxxxx",
  "version": "6.1.0",
  "message": "greetings from podinfo v6.1.0",
  ...
}

Test Results

Our testing confirmed the following:

Creating a Canary resource without the hosts field was successful
The HTTPRoute was generated correctly and reached the Accepted status
We could access the application through the Gateway
After updating the Deployment, the canary release progressed normally and eventually completed promotion
We could access the new version (v6.1.0) after the update

This confirms that the hosts field in Flagger's Canary resources is not mandatory and can be omitted without affecting functionality.

Conclusion

When integrating Flagger with the Gateway API, the hosts field is optional. If you don't need routing based on specific hostnames, you can simply specify gatewayRefs to achieve canary releases with a simpler configuration.

References

Repository with the test setup described in this article

Troubleshooting Docker credsStore Auto-Configuration Issues in VS Code Dev Containers

suin — Thu, 26 Dec 2024 06:31:49 +0000

When using VS Code's Dev Container extension, you might encounter credentials-related errors while trying to access registries through docker pull, az acr login, or other CLI tools. These errors might look like this:

error getting credentials - err: exit status 255, out: ``

You might also experience failures with commands like kcl mod add or az acr login, or encounter 403 errors related to authentication. This article explains the cause and solutions to these issues.

Note about "Host Machine" in this article

In Dev Containers, the "host machine" refers to the machine running VS Code.

Even if your Docker daemon is running on a cloud or remote server and you're only running VS Code on your local Mac or PC, we'll refer to the VS Code side (local PC) = host machine in this context.

The Problem

Commands like kcl mod add or docker pull succeed when executed from VS Code's terminal inside the container, but fail with error getting credentials when accessing the container through SSH or docker exec.
Looking at ~/.docker/config.json inside the container, you'll find credsStore is set to something like dev-containers-xxxxxx....
Removing this credsStore temporarily fixes the error, but it reappears when reconnecting to the Dev Container in VS Code.

Root Cause

Understanding Docker Credential Helper and VS Code Dev Containers Extension

VS Code's Dev Container extension includes a feature that automatically sets up a Docker Credential Helper. This mechanism attempts to share registry authentication information with the container using the ~/.docker/config.json and credsStore from the host machine running VS Code, not from within the container.

When starting a Dev Container, if there's no credsStore specified in the container's ~/.docker/config.json, and Dev Container settings are default (or dev.containers.dockerCredentialHelper is enabled), the VS Code Dev Containers extension automatically inserts a credsStore.
This results in the following content in the container's ~/.docker/config.json:

{
  "credsStore": "dev-containers-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
}

Consequently, when running docker pull or az acr login inside the container, it attempts to call a helper like docker-credential-dev-containers-xxxxxx.
- This helper is a generated script inside the container that performs IPC (Inter-Process Communication) with VS Code (Dev Container extension) to query authentication information tied to the host machine's ~/.docker/config.json.

The REMOTE_CONTAINERS_IPC Environment Variable

The success of this authentication depends on the REMOTE_CONTAINERS_IPC environment variable.

When attaching to a container through VS Code's terminal, the Dev Container extension automatically sets REMOTE_CONTAINERS_IPC, allowing successful authentication via credsStore for commands like docker pull.
However, when accessing the container through SSH or docker exec without VS Code, this environment variable isn't set, causing the docker-credential-xxx to fail its IPC communication, resulting in the "error getting credentials - err: exit status 255" error.

Solutions

1. Remove `credsStore` from the Container's `~/.docker/config.json`

Opening ~/.docker/config.json inside the container and removing the credsStore line will temporarily allow docker pull and similar commands to succeed.
Example:

# Inside container: ~/.docker/config.json
{
  // "credsStore": "dev-containers-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
  "auths": { ... }
}

However, this isn't a permanent solution if VS Code settings remain default, as credsStore will be re-added on each startup.

2. Disable `dev.containers.dockerCredentialHelper`

In your Dev Container configuration file (devcontainer.json), you can prevent the Docker Credential Helper from being installed by setting dev.containers.dockerCredentialHelper: false:

// devcontainer.json example
{
  "customizations": {
    "vscode": {
      "settings": {
        // Set this to false
        "dev.containers.dockerCredentialHelper": false
      }
    }
  }
}

This prevents automatic credsStore configuration in the container's ~/.docker/config.json, eliminating IPC errors. However, you'll lose the ability to inherit Docker authentication from the host machine, requiring separate docker login commands in the container and consideration of associated security risks.

3. Enable `REMOTE_CONTAINERS_IPC` for SSH/Alternative Shells

If you want to maintain IPC functionality even when accessing the container without VS Code, you can persist the REMOTE_CONTAINERS_IPC value set by VS Code across shells.

For example, using Dev Container's postAttachCommand, you could write the environment variable to ~/.config/fish/conf.d/ (for fish shell):

if [ -n "${REMOTE_CONTAINERS_IPC+x}" ]; then
  # Export environment variable for fish shell
  echo "set -x REMOTE_CONTAINERS_IPC ${REMOTE_CONTAINERS_IPC}" > ~/.config/fish/conf.d/remote-containers-ipc.fish
fi

This allows recognition of REMOTE_CONTAINERS_IPC even when accessing via SSH. However, since REMOTE_CONTAINERS_IPC might have random values per connection, you'll need to establish operational rules.

4. Clean Up Host Machine Credentials

Authentication issues like 403 errors during docker pull might occur if credential information in the host machine's ~/.docker/config.json or key management (like Keychain Access) is corrupted.

On macOS, try deleting credentials for ghcr.io, quay.io, etc., from Keychain Access and then run docker login again.
Verify that tokens obtained after docker login have necessary permissions like packages:read.

Reviewing these aspects can help prevent 403 errors (insufficient permissions).

Summary

VS Code's Dev Container extension includes a Docker Credential Helper that inherits Docker authentication from the host machine.
- Here, "host machine" means the machine running VS Code, not the Docker daemon host.
This mechanism automatically configures credsStore in the container's ~/.docker/config.json, but can fail authentication in environments without REMOTE_CONTAINERS_IPC (like SSH).
Available solutions include:
1. Removing credsStore from the container's ~/.docker/config.json (though it may be re-added)
2. Disabling the Credential Helper by setting dev.containers.dockerCredentialHelper to false
3. Implementing a custom mechanism to persist REMOTE_CONTAINERS_IPC across external shells
4. Cleaning up Docker credentials on the host machine (checking/re-logging via Keychain Access)

References

Devcontainer version 0.275 - auto adding ~/.docker/config.json with credsStore specified which cause az acr login to fail · Issue #7982 · microsoft/vscode-remote-release

Deploying Helm Charts to Multiple Kubernetes Clusters with Cluster API's HelmChartProxy

suin — Thu, 21 Nov 2024 09:59:49 +0000

In this article, I'll introduce you to HelmChartProxy, a powerful feature of Cluster API that enables bulk deployment of Helm charts across multiple Kubernetes clusters.

Introduction

When managing multiple Kubernetes clusters, you often need to deploy the same Helm charts across different environments. Traditionally, this meant running Helm commands individually for each cluster. However, with Cluster API's HelmChartProxy, you can now streamline this process with bulk deployments.

We'll explore:

Bulk deployment to all clusters
Targeted deployment using labels
Deployment from private registries

Setting Up the Test Environment

Prerequisites

Docker (16GB+ memory recommended)
kubectl
kind
clusterctl

1. Creating the Management Cluster

First, let's create a management cluster using kind. Create the following configuration file:

# kind-cluster-with-extramounts.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
  ipFamily: dual
nodes:
- role: control-plane
  extraMounts:
    - hostPath: /var/run/docker.sock
      containerPath: /var/run/docker.sock

Create the cluster using this configuration:

kind create cluster --config kind-cluster-with-extramounts.yaml

2. Setting Up Cluster API

Install Cluster API on the management cluster:

export CLUSTER_TOPOLOGY=true
clusterctl init --infrastructure docker --addon helm

Wait for all components to be ready:

kubectl wait --for=condition=Available --timeout=300s -n capi-system deployments --all && \
kubectl wait --for=condition=Available --timeout=300s -n capi-kubeadm-bootstrap-system deployments --all && \
kubectl wait --for=condition=Available --timeout=300s -n capi-kubeadm-control-plane-system deployments --all && \
kubectl wait --for=condition=Available --timeout=300s -n capd-system deployments --all && \
kubectl wait --for=condition=Available --timeout=300s -n caaph-system deployments --all

3. Creating Workload Clusters

We'll create two clusters named "muscat" and "delaware":

# Generate manifest for muscat cluster
clusterctl generate cluster muscat \
  --flavor development \
  --kubernetes-version v1.28.0 \
  --control-plane-machine-count=3 \
  --worker-machine-count=3 \
  > muscat.yaml

# Generate manifest for delaware cluster
clusterctl generate cluster delaware \
  --flavor development \
  --kubernetes-version v1.28.0 \
  --control-plane-machine-count=3 \
  --worker-machine-count=3 \
  > delaware.yaml

# Create the clusters
kubectl apply -f muscat.yaml
kubectl apply -f delaware.yaml

Get the kubeconfig files:

clusterctl get kubeconfig muscat > muscat-kubeconfig.yaml
clusterctl get kubeconfig delaware > delaware-kubeconfig.yaml

4. Setting Up CNI

Install Calico on both clusters:

# Install Calico on muscat
kubectl --kubeconfig=muscat-kubeconfig.yaml apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.26.1/manifests/calico.yaml

# Install Calico on delaware
kubectl --kubeconfig=delaware-kubeconfig.yaml apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.26.1/manifests/calico.yaml

Deploying with HelmChartProxy

Deploying to All Clusters

Let's look at an example of deploying nginx to all clusters:

# helm-chart-proxy-nginx.yaml
apiVersion: addons.cluster.x-k8s.io/v1alpha1
kind: HelmChartProxy
metadata:
  name: nginx
spec:
  clusterSelector: {} # Empty selector means "select all clusters"
  repoURL: oci://registry-1.docker.io/bitnamicharts
  chartName: nginx
  version: "18.2.5"
  releaseName: nginx
  namespace: nginx
  options:
    waitForJobs: true
    atomic: true
    wait: true
    timeout: 5m
    install:
      createNamespace: true
  valuesTemplate: |-
    service:
      type: NodePort

Deploy and verify:

# Deploy nginx
kubectl apply -f helm-chart-proxy-nginx.yaml

# Wait for HelmReleaseProxy to be ready
kubectl wait --for=condition=Ready --timeout=300s helmreleaseproxy --all

# Check pods on muscat
kubectl --kubeconfig=muscat-kubeconfig.yaml get pods -n nginx

# Check pods on delaware
kubectl --kubeconfig=delaware-kubeconfig.yaml get pods -n nginx

Targeted Deployment Using Labels

You can deploy to specific clusters using labels:

# helm-chart-proxy-nginx.yaml
apiVersion: addons.cluster.x-k8s.io/v1alpha1
kind: HelmChartProxy
metadata:
  name: nginx
spec:
  clusterSelector:
    matchLabels:
      use-nginx: "true"  # Only deploy to clusters with this label
  # ... (other settings remain the same)

Usage:

# Deploy nginx
kubectl apply -f helm-chart-proxy-nginx.yaml

# Label the muscat cluster
kubectl label cluster muscat use-nginx=true

# Check deployment status
kubectl get helmreleaseproxy -A

# Verify nginx is running on muscat
kubectl --kubeconfig=muscat-kubeconfig.yaml get pods -n nginx

# Verify nginx is not running on delaware
kubectl --kubeconfig=delaware-kubeconfig.yaml get pods -n nginx

Deploying from Private Registries

Let's use GitHub Container Registry as an example:

Create a GitHub Personal Access Token (needs read:packages permission)
Set up registry credentials:

# Create config.json with base64 encoded credentials
echo -n "YOUR_GITHUB_USERNAME:YOUR_GITHUB_TOKEN" | base64 > auth.txt

cat > config.json << EOF
{
  "auths": {
    "ghcr.io": {
      "auth": "$(cat auth.txt)"
    }
  }
}
EOF

# Create the secret
kubectl create secret generic github-creds \
  --from-file=config.json \
  -n caaph-system

Create HelmChartProxy for private charts:

apiVersion: addons.cluster.x-k8s.io/v1alpha1
kind: HelmChartProxy
metadata:
  name: private-chart
spec:
  clusterSelector: {}
  repoURL: oci://ghcr.io/YOUR_GITHUB_USERNAME
  chartName: YOUR_CHART_NAME
  version: "0.1.0"
  credentials:
    secret:
      name: github-creds
      namespace: caaph-system
    key: config.json
  # ... (other settings)

Conclusion

HelmChartProxy significantly simplifies the process of deploying Helm charts across multiple clusters. Key benefits include:

Reduced operational overhead through bulk deployments
Flexible deployment control using labels
Support for private registries

If you're managing multiple Kubernetes clusters, consider incorporating HelmChartProxy into your workflow. It can greatly streamline your Helm chart deployment process and make your multi-cluster management more efficient.

How to Host Helm Charts on GitHub Container Registry

suin — Thu, 21 Nov 2024 07:14:40 +0000

Recently, I've been working more with application management in Kubernetes environments, which led me to think deeply about Helm chart management. Today, I'll show you how to host Helm charts using GitHub Container Registry (GHCR). What makes GHCR particularly attractive is its free private registry feature, making it an ideal choice for personal projects or small team collaborations.

Prerequisites

You'll need:

Helm
A GitHub account with a Classic Personal Access Token
- Your token needs the write:packages scope
- Note: Fine-grained permissions aren't currently supported for GHCR

Our Example Chart

For this tutorial, we'll use a minimalist chart that creates a hello-world namespace. Here's our chart structure:

charts/hello-world/
├── Chart.yaml
├── templates/
│   └── namespace.yaml

# Chart.yaml
apiVersion: v2
name: hello-world
description: A simple Helm chart that creates hello-world namespace
type: application
version: 0.1.0
appVersion: "1.0.0"

# templates/namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: hello-world

Publishing to GHCR

1. Log into GitHub Container Registry

First, log in using the Helm CLI:

helm registry login ghcr.io -u YOUR_GITHUB_USERNAME

When prompted, enter your Personal Access Token (PAT) as the password.

2. Package Your Chart

helm package charts/hello-world

This creates a hello-world-0.1.0.tgz file.

3. Push to GHCR

helm push hello-world-0.1.0.tgz oci://ghcr.io/YOUR_GITHUB_USERNAME

That's it! Simple and straightforward.

Using Your Published Chart

To install the chart from GHCR:

helm install hello-world oci://ghcr.io/YOUR_GITHUB_USERNAME/hello-world --version 0.1.0

Cleanup

After testing, you can clean up with these commands:

# Uninstall the Helm release
helm uninstall hello-world

# Delete the created namespace
kubectl delete namespace hello-world

# Remove the local package file
rm hello-world-0.1.0.tgz

Why GHCR?

GitHub Container Registry offers several advantages:

Free private registries
Seamless GitHub integration for easier management
OCI standard compatibility

Additionally, having your Helm charts and source code in the same ecosystem simplifies your DevOps workflow. The ability to manage access through GitHub's familiar interface is a significant bonus for teams already using GitHub for their development work.

Wrapping Up

Hosting Helm charts on GitHub Container Registry is a straightforward process that provides a robust solution for chart management. Whether you're working on personal projects or collaborating with a small team, GHCR offers a reliable, cost-effective platform for your Helm charts.

Give it a try and let me know how it works for your use case! Feel free to share your experiences or ask questions in the comments below.

How to Use a GitHub Private Repository as a Helm Chart Repository

suin — Thu, 21 Nov 2024 06:03:46 +0000

In this article, I'll show you how to create your own private Helm Chart repository using a GitHub private repository. This approach is perfect for personal or team use.

Why Use GitHub as a Helm Chart Repository?

While this setup might not be ideal for production environments, it's excellent for development and testing purposes because:

You can securely manage private Kubernetes manifests
It's easy to share Helm Charts within your team
No additional infrastructure management is required

Prerequisites

You'll need:

A GitHub private repository
A GitHub Personal Access Token
- For Classic Token: requires repo scope
- For Fine-grained Token: needs read access to "Contents"
Helm CLI installed
git command line tool

Sample Chart Overview

For this tutorial, we'll use a minimal hello-world chart. Here's what it looks like:

# Chart.yaml
apiVersion: v2
name: hello-world
description: A simple Helm chart that creates hello-world namespace
type: application
version: 0.1.0
appVersion: "1.0.0"

# templates/namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: hello-world

This chart is intentionally simple:

It creates a namespace called hello-world
It has no configurable values
When deployed, it creates a new namespace in your Kubernetes cluster

While real-world charts are typically more complex, we're keeping it minimal to focus on the repository setup process.

Step-by-Step Guide

1. Package Your Chart

First, let's package our hello-world chart:

cd private-repo
helm package hello-world/

This creates hello-world-0.1.0.tgz, containing our Chart.yaml and templates/namespace.yaml.

2. Generate the Index File

Create an index file to tell Helm about available charts:

helm repo index .

This generates index.yaml with content like:

apiVersion: v1
entries:
  hello-world:
    - apiVersion: v2
      appVersion: 1.0.0
      created: "2024-11-21T10:00:00.000000000Z"
      description: A simple Helm chart that creates hello-world namespace
      digest: 1234567890abcdef... # actual hash will vary
      name: hello-world
      type: application
      urls:
        - hello-world-0.1.0.tgz
      version: 0.1.0

3. Push to GitHub

Upload everything to GitHub:

git init
git add .
git commit -s -m "Initial commit"
git branch -M main
git remote add origin git@github.com:your-username/repo-name.git
git push -u origin main

4. Configure Helm

Add the repository to your local Helm configuration:

helm repo add --username your-github-username --password your-github-token private-repo 'https://raw.githubusercontent.com/your-username/repo-name/main'

Update the repository information:

helm repo update

5. Verify Setup

Check if your chart is discoverable:

helm search repo private-repo/hello-world

You should see something like:

NAME                        CHART VERSION   APP VERSION   DESCRIPTION
private-repo/hello-world    0.1.0          1.0.0        A simple Helm chart that creates hello-world namespace

Using Your Chart

Before installing, it's good practice to do a dry-run:

helm install test-hello private-repo/hello-world --dry-run

Output:

# Source: hello-world/templates/namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: hello-world

If everything looks good, proceed with the installation:

helm install hello-world private-repo/hello-world

Successful installation output:

NAME: hello-world
LAST DEPLOYED: Thu Nov 21 14:45:40 2024
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None

Verify:

kubectl get namespace hello-world

Repository Structure

Your final repository structure should look like this:

.
├── README.md
├── index.yaml              # Helm repository index
├── hello-world-0.1.0.tgz   # Packaged chart
└── hello-world/           # Chart source
    ├── Chart.yaml         # Chart metadata
    └── templates/         # Kubernetes manifest templates
        └── namespace.yaml # Namespace manifest

That's it! You now have your own private Helm Chart repository. This setup is perfect for sharing charts within your team and managing versions effectively.

Feel free to reach out in the comments if you have any questions or run into issues!

Getting Started with Cluster API Locally: A Developer's Guide to CAPD

suin — Wed, 20 Nov 2024 06:54:27 +0000

Interested in Cluster API for automating Kubernetes cluster management but hesitant because you think you need a cloud environment? Good news! You can actually try Cluster API right on your local machine with just Docker. In this guide, we'll walk through setting up a testing environment using CAPD (Cluster API Provider Docker).

Prerequisites

You'll need the following tools installed:

Docker
kubectl
kind
clusterctl

Overview

Here's what we'll cover:

Creating a management cluster using kind
Initializing Cluster API (CAPD)
Creating a workload cluster
Setting up CNI

Let's dive in!

1. Creating the Management Cluster

First, we'll create a management cluster using kind that will serve as the foundation for CAPD.

Create a configuration file that enables access to the host's Docker socket:

# kind-cluster-with-extramounts.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
  ipFamily: dual
nodes:
- role: control-plane
  extraMounts:
    - hostPath: /var/run/docker.sock
      containerPath: /var/run/docker.sock

Now, create the kind cluster using this configuration:

kind create cluster --config kind-cluster-with-extramounts.yaml

You should see output similar to this:

Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.31.0) 🖼
 ✓ Preparing nodes 📦  
 ✓ Writing configuration 📜 
 ✓ Starting control-plane 🕹️ 
 ✓ Installing CNI 🔌 
 ✓ Installing StorageClass 💾 
Set kubectl context to "kind-kind"

2. Initializing Cluster API

Next, let's install Cluster API on our management cluster:

export CLUSTER_TOPOLOGY=true && clusterctl init --infrastructure docker

This command installs all necessary components. If successful, you'll see:

Fetching providers
Installing cert-manager version="v1.16.0"
Waiting for cert-manager to be available...
Installing provider="cluster-api" version="v1.8.5" targetNamespace="capi-system"
Installing provider="bootstrap-kubeadm" version="v1.8.5" targetNamespace="capi-kubeadm-bootstrap-system"
Installing provider="control-plane-kubeadm" version="v1.8.5" targetNamespace="capi-kubeadm-control-plane-system"
Installing provider="infrastructure-docker" version="v1.8.5" targetNamespace="capd-system"
Your management cluster has been initialized successfully!

3. Creating a Workload Cluster

Now let's create a Kubernetes cluster for testing. We'll name it "muscat" 🍇:

clusterctl generate cluster muscat \
  --flavor development \
  --kubernetes-version v1.31.0 \
  --control-plane-machine-count=3 \
  --worker-machine-count=3 \
  > muscat.yaml
kubectl apply -f muscat.yaml

Check the cluster status:

kubectl get cluster

NAME     CLUSTERCLASS   PHASE         AGE     VERSION
muscat   quick-start    Provisioned   5m47s   v1.31.0

4. Setting Up CNI

Finally, let's enable networking functionality by installing Calico:

# First, get the kubeconfig
clusterctl get kubeconfig muscat > kubeconfig.muscat.yaml
# Install Calico
kubectl --kubeconfig=./kubeconfig.muscat.yaml apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.27.0/manifests/calico.yaml

After a few moments, all nodes should be in Ready state:

kubectl --kubeconfig=./kubeconfig.muscat.yaml get nodes
NAME                            STATUS   ROLES           AGE     VERSION
muscat-md-0-9mhvz-4xxcd-42nh8   Ready    <none>          4m20s   v1.31.0
muscat-md-0-9mhvz-4xxcd-8hghp   Ready    <none>          4m25s   v1.31.0
muscat-md-0-9mhvz-4xxcd-mxg7k   Ready    <none>          4m20s   v1.31.0
muscat-r65sn-592c8              Ready    control-plane   3m35s   v1.31.0
muscat-r65sn-xrzfl              Ready    control-plane   4m41s   v1.31.0
muscat-worker-08sx08            Ready    <none>          4m17s   v1.31.0
muscat-worker-u6l39f            Ready    <none>          4m17s   v1.31.0
muscat-worker-ydhg40            Ready    <none>          4m17s   v1.31.0

Conclusion

You now have a fully functional Cluster API testing environment on your local machine! Here's what we've accomplished:

A management cluster (kind) running Cluster API
A workload cluster with three control plane nodes and three worker nodes
All this running locally with just Docker - no cloud provider needed!

You're now ready to start experimenting with various Cluster API features in this local environment. Happy clustering! 🎉

Sharing Secrets Between Kubernetes Clusters Using external-secrets PushSecret

suin — Thu, 14 Nov 2024 07:12:41 +0000

In this article, I'll explain how to share secrets between Kubernetes clusters using the PushSecret feature of external-secrets. In multi-cluster environments, secret synchronization is one of the crucial operational challenges. By building a system that securely shares and automatically synchronizes secrets between clusters, we can improve both operational efficiency and security.

Desired Architecture

We'll implement the following architecture to automatically synchronize secrets from the source cluster to the target cluster:

Prerequisites

To follow along with this article, you'll need the following tools:

Docker
k3d
kubectl
Helm
kubectx (optional: for easier context switching)
kubectl-view-secret (optional: for viewing secret contents)

Environment Setup

We'll set up two clusters using k3d:

Source cluster: k3d-source-cluster
Target cluster: k3d-target-cluster

Creating a Shared Network

First, let's create a shared network to enable communication between clusters. This network will facilitate communication between the two Kubernetes clusters we'll create later.

docker network create shared-net --subnet 172.28.0.0/16 --gateway 172.28.0.1

This command creates a network with the following characteristics:

Subnet: 172.28.0.0/16
Gateway: 172.28.0.1
Network name: shared-net

Setting Up the Source Cluster

The source cluster will provide the secrets. Create a source-cluster.yaml with the following content:

apiVersion: k3d.io/v1alpha5
kind: Simple
metadata:
  name: source-cluster
servers: 1
agents: 1
image: docker.io/rancher/k3s:v1.30.0-k3s1
kubeAPI:
  host: 0.0.0.0
  hostIP: 127.0.0.1
  hostPort: "6443"
ports:
  - port: 8080:80
    nodeFilters:
      - loadbalancer
registries:
  create:
    name: registry.localhost
    host: 127.0.0.1
    hostPort: "15000"
network: shared-net
options:
  k3d:
    wait: true
  kubeconfig:
    updateDefaultKubeconfig: true
    switchCurrentContext: true
  k3s:
    extraArgs:
      # Different CIDR ranges to avoid overlap
      - arg: "--cluster-cidr=10.42.0.0/16"
        nodeFilters:
          - server:*
      - arg: "--service-cidr=10.43.0.0/16"
        nodeFilters:
          - server:*

Key points of this configuration:

Network Settings
- KubeAPI: Bound to port 6443
- Load balancer: Port mapping 8080:80
- Network: Uses the shared network created earlier
Registry Settings
- Creates a local registry (name: registry.localhost)
- Bound to port 15000
CIDR Settings
- Cluster CIDR: 10.42.0.0/16
- Service CIDR: 10.43.0.0/16 (Important to avoid network overlaps between clusters)

Create the cluster with:

k3d cluster create --config source-cluster.yaml

Setting Up the Target Cluster

The target cluster will receive the secrets. Create a target-cluster.yaml with different settings:

apiVersion: k3d.io/v1alpha5
kind: Simple
metadata:
  name: target-cluster
servers: 1
agents: 1
image: docker.io/rancher/k3s:v1.30.0-k3s1
kubeAPI:
  host: 0.0.0.0
  hostIP: 127.0.0.1
  hostPort: "6444"  # Different from source cluster
ports:
  - port: 8081:80   # Different from source cluster
    nodeFilters:
      - loadbalancer
registries:
  use:
    - registry.localhost  # Use source cluster's registry
network: shared-net
options:
  k3d:
    wait: true
  kubeconfig:
    updateDefaultKubeconfig: true
    switchCurrentContext: true
  k3s:
    extraArgs:
      # Different CIDR ranges from source cluster
      - arg: "--cluster-cidr=10.44.0.0/16"
        nodeFilters:
          - server:*
      - arg: "--service-cidr=10.45.0.0/16"
        nodeFilters:
          - server:*

Key differences from the source cluster:

Different Ports
- KubeAPI port: 6444 (source uses 6443)
- Load balancer port: 8081:80 (source uses 8080:80)
- CIDR ranges: 10.44.0.0/16 and 10.45.0.0/16
Registry Setup
- Uses the registry created by source cluster
- Specified in the use section
Network Settings
- Uses the same shared network
- Enables inter-cluster communication

Create the cluster:

k3d cluster create --config target-cluster.yaml

Setting Up external-secrets

Now that our clusters are ready, let's set up external-secrets.

Installing on the Source Cluster

# Switch to source cluster context
kubectl config use-context k3d-source-cluster

helm repo add external-secrets https://charts.external-secrets.io
helm repo update

helm install external-secrets \
   external-secrets/external-secrets \
    -n external-secrets \
    --create-namespace

Configuring Target Cluster Authentication

Let's set up the authentication credentials needed for the source cluster to access the target cluster.

Getting Authentication Information

First, get the target cluster's client certificate information:

kubectl config view --raw

# Note down these values:
# - client-certificate-data
# - client-key-data
# - certificate-authority-data

Setting Up Authentication

Create target-cluster-credentials.yaml:

apiVersion: v1
kind: Secret
metadata:
  name: target-cluster-credentials
  namespace: default
type: Opaque
data:
  client-certificate-data: ... # Base64 encoded data from kubeconfig
  client-key-data: ... # Base64 encoded data from kubeconfig

Apply the credentials to the source cluster:

# Switch to source cluster context
kubectl config use-context k3d-source-cluster

# Apply credentials
kubectl apply -f target-cluster-credentials.yaml

# Verify the created Secret
kubectl get secret target-cluster-credentials -o yaml

Setting Up SecretStore

The SecretStore defines the backend store where external-secrets will store and retrieve secrets.

Create secret-store.yaml:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: target-cluster
  namespace: default
spec:
  provider:
    kubernetes:
      remoteNamespace: default  # Target cluster namespace
      server:
        url: https://k3d-target-cluster-server-0:6443  # Using k3d internal hostname
        caBundle: ... # certificate-authority-data from kubeconfig
      auth:
        cert:
          clientCert:
            name: target-cluster-credentials
            key: client-certificate-data
          clientKey:
            name: target-cluster-credentials
            key: client-key-data

Apply and verify the SecretStore:

# Apply SecretStore
kubectl apply -f secret-store.yaml

# Check status
kubectl describe secretstore target-cluster

# Expected output should include:
Status:
  Conditions:
    Last Transition Time:  ...
    Message:               SecretStore validated
    Reason:               Valid
    Status:               True
    Type:                 Ready

Setting Up and Testing PushSecret

Creating PushSecret

PushSecret automatically pushes secrets from the source cluster to the target cluster.

Create a Sample Secret (Source Cluster)

# Create sample secret
kubectl create secret generic my-secret \
  --from-literal=username=admin \
  --from-literal=password=supersecret

# Verify created secret
kubectl get secret my-secret -o yaml

Create PushSecret Definition

Create push-secret.yaml:

apiVersion: external-secrets.io/v1alpha1
kind: PushSecret
metadata:
  name: pushsecret-example
  namespace: default
spec:
  # Replace existing secrets in provider
  updatePolicy: Replace
  # Delete provider secret when PushSecret is deleted
  deletionPolicy: Delete
  # Resync interval
  refreshInterval: 10s
  # SecretStore to push secrets to
  secretStoreRefs:
    - name: target-cluster
      kind: SecretStore
  # Target Secret
  selector:
    secret:
      name: my-secret  # Source cluster Secret name
  data:
    - match:
        secretKey: username  # Source cluster Secret key
        remoteRef:
          remoteKey: my-secret-copy  # Target cluster Secret name
          property: username-copy    # Target cluster Secret key
    - match:
        secretKey: password  # Source cluster Secret key
        remoteRef:
          remoteKey: my-secret-copy  # Target cluster Secret name
          property: password-copy    # Target cluster Secret key

Apply PushSecret:

kubectl apply -f push-secret.yaml

# Check status
kubectl describe pushsecret pushsecret-example

Verifying Operation

Check Secret in Target Cluster

# Switch to target cluster
kubectl config use-context k3d-target-cluster

# Check secret
kubectl describe secret my-secret-copy

# Expected output:
Name:         my-secret-copy
Namespace:    default
Labels:       <none>
Annotations:  <none>

Type:  Opaque

Data
====
password-copy:  11 bytes
username-copy:  5 bytes

Verify Secret Contents

# Using kubectl-view-secret plugin
kubectl-view-secret my-secret-copy --all

# Or directly using base64 decode
kubectl get secret my-secret-copy -o jsonpath='{.data.username-copy}' | base64 -d
kubectl get secret my-secret-copy -o jsonpath='{.data.password-copy}' | base64 -d

Expected output:

password-copy='supersecret'
username-copy='admin'

Test Automatic Synchronization

Update the secret in the source cluster and verify it's reflected in the target cluster:

# Switch to source cluster
kubectl config use-context k3d-source-cluster

# Update secret
kubectl create secret generic my-secret \
  --from-literal=username=newadmin \
  --from-literal=password=newsecret \
  --dry-run=client -o yaml | kubectl apply -f -

# Switch to target cluster and verify
kubectl config use-context k3d-target-cluster
kubectl-view-secret my-secret-copy --all

Cleanup

# Switch to source cluster
kubectl config use-context k3d-source-cluster

# Delete PushSecret
kubectl delete pushsecret pushsecret-example

# Verify secret deletion in target cluster
kubectl config use-context k3d-target-cluster
kubectl get secret my-secret-copy
# Should show: "Error from server (NotFound): secrets "my-secret-copy" not found"

# Delete clusters
k3d cluster delete --config source-cluster.yaml
k3d cluster delete --config target-cluster.yaml

# Delete shared network
docker network rm shared-net

Conclusion

In this article, we've explored how to use external-secrets' PushSecret feature to share secrets between Kubernetes clusters. In production environments, this feature can significantly improve secret management efficiency in multi-cluster setups!

References

Sync Kubernetes Secrets to AWS Secrets Manager Using external-secrets PushSecret

suin — Thu, 14 Nov 2024 01:25:54 +0000

When managing sensitive information in Kubernetes, you can use an operator called external-secrets to integrate with external secret providers like AWS Secrets Manager.

While the common usage pattern is to synchronize sensitive information stored in AWS Secrets Manager as Kubernetes Secrets, this article introduces PushSecret, which enables reverse synchronization - pushing Kubernetes Secrets to AWS Secrets Manager.

Let's explore the basic usage of PushSecret.

Prerequisites

Kubernetes cluster
AWS account
kubectl
Helm
AWS CLI

Installing external-secrets

First, install external-secrets using Helm:

helm repo add external-secrets https://charts.external-secrets.io

helm install external-secrets \
   external-secrets/external-secrets \
    -n external-secrets \
    --create-namespace

Setting Up AWS Credentials

Set AWS credentials as environment variables and verify they are configured correctly:

export AWS_ACCESS_KEY_ID=xxxx
export AWS_SECRET_ACCESS_KEY=xxxx
export AWS_REGION=ap-northeast-1

# Verify credentials
aws sts get-caller-identity

Next, store these credentials as a Kubernetes Secret:

kubectl create secret generic aws-credentials \
  --from-literal=access-key-id=${AWS_ACCESS_KEY_ID} \
  --from-literal=secret-access-key=${AWS_SECRET_ACCESS_KEY}

Configuring SecretStore

Create a SecretStore to connect to AWS Secrets Manager:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-secretsmanager
  namespace: default
spec:
  provider:
    aws:
      service: SecretsManager
      region: ap-northeast-1
      auth:
        secretRef:
          accessKeyIDSecretRef:
            name: aws-credentials
            key: access-key-id
          secretAccessKeySecretRef:
            name: aws-credentials
            key: secret-access-key

kubectl apply -f secret-store.yaml

Configuring PushSecret

Create a PushSecret to synchronize Kubernetes Secrets to AWS Secrets Manager:

apiVersion: external-secrets.io/v1alpha1
kind: PushSecret
metadata:
  name: pushsecret-example
  namespace: default
spec:
  # Overwrite existing secrets in provider during sync
  updatePolicy: Replace
  # Delete provider secrets when PushSecret is deleted
  deletionPolicy: Delete
  # Resync interval
  refreshInterval: 10s
  # SecretStore to push secrets to
  secretStoreRefs:
    - name: aws-secretsmanager
      kind: SecretStore
  # Target Secret for synchronization
  selector:
    secret:
      name: my-secret
  # Key configuration for synchronization
  data:
    - match:
        secretKey: foo  # Secret key
        remoteRef:
          remoteKey: my-secret-foo  # AWS Secrets Manager secret name
      metadata:
        secretPushFormat: string

kubectl apply -f push-secret.yaml

Verification

1. Creating a Secret

First, create the target Secret:

kubectl create secret generic my-secret \
  --from-literal=foo=bar

Verify that it has been synchronized to AWS Secrets Manager:

aws secretsmanager get-secret-value \
  --secret-id my-secret-foo

If successful, you should see a response like this:

{
    "ARN": "arn:aws:secretsmanager:ap-northeast-1:000000000000:secret:my-secret-foo-rUBCkr",
    "Name": "my-secret-foo",
    "VersionId": "00000000-0000-0000-0000-000000000001",
    "SecretString": "bar",
    "VersionStages": [
        "AWSCURRENT"
    ],
    "CreatedDate": "2024-11-14T09:50:34.787000+09:00"
}

2. Updating the Secret

Let's update the Secret value:

kubectl create secret generic my-secret \
  --from-literal=foo=baz \
  --dry-run=client -o yaml | kubectl apply -f -

Verify that the value has been updated in AWS Secrets Manager:

aws secretsmanager get-secret-value \
  --secret-id my-secret-foo

You can confirm that SecretString has changed from bar to baz:

{
    "ARN": "arn:aws:secretsmanager:ap-northeast-1:000000000000:secret:my-secret-foo-rUBCkr",
    "Name": "my-secret-foo",
    "VersionId": "00000000-0000-0000-0000-000000000002",
    "SecretString": "baz",
    "VersionStages": [
        "AWSCURRENT"
    ],
    "CreatedDate": "2024-11-14T10:03:41.913000+09:00"
}

3. Verifying Deletion Behavior

Delete the Secret:

kubectl delete secret my-secret

At this point, the secret in AWS Secrets Manager is not deleted.

Next, delete the PushSecret:

kubectl delete pushsecret pushsecret-example

This operation will also delete the secret from AWS Secrets Manager:

aws secretsmanager list-secrets

{
    "SecretList": []
}

Cleanup

When you delete the external-secrets PushSecret, my-secret-foo remains in AWS Secrets Manager as "scheduled for deletion". To immediately delete the secret from AWS Secrets Manager:

aws secretsmanager delete-secret \
  --secret-id my-secret-foo \
  --force-delete-without-recovery

Delete external-secrets:

helm uninstall external-secrets -n external-secrets

Delete AWS credentials:

kubectl delete secret aws-credentials

Summary

We've seen how external-secrets' PushSecret can synchronize Kubernetes Secrets to AWS Secrets Manager. This feature can be useful in several scenarios:

Sharing sensitive information managed in Kubernetes with other AWS services
Sharing Secrets between Kubernetes clusters via AWS Secrets Manager
Backing up Kubernetes Secrets to AWS Secrets Manager

While this example used external-secrets with AWS Secrets Manager, there are various other providers available for SecretStore. One particularly interesting provider is Kubernetes itself. I'm personally interested in trying out direct Secret synchronization between Kubernetes clusters and plan to write about that experience in a future article.

Reference Links

Securing external-dns: Encrypting TXT Registry Records

suin — Wed, 06 Nov 2024 07:55:40 +0000

In my previous article, we explored how to automate DNS record management using external-dns with AWS Route53. We briefly mentioned that management information stored in TXT records is publicly visible. Today, let's dive into how to secure this information using external-dns's TXT record encryption feature.

Understanding TXT Registry in external-dns

external-dns stores management information in what's called a Registry. While multiple Registry options exist, including DynamoDB and AWS Service Discovery, TXT Registry is particularly interesting because it avoids cloud vendor lock-in. However, since TXT records are publicly accessible, encrypting them adds an important security layer while maintaining the benefits of vendor independence.

Prerequisites

A configured Kubernetes cluster
Required command-line tools installed:
- kubectl
- helm
- aws
- dig
AWS account access and secret keys

Implementation Steps

1. Creating an AWS Route53 Hosted Zone

First, set your AWS credentials as environment variables:

export AWS_ACCESS_KEY_ID=AKIAXXXXXXXXXXXXXXXX
export AWS_SECRET_ACCESS_KEY=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Verify your AWS credentials:

aws sts get-caller-identity

Create a Hosted Zone:

aws route53 create-hosted-zone --name example-tutorial.com --caller-reference external-dns-tutorial-$(date +%s)

Note: The caller-reference must be unique for each creation attempt. We're using a timestamp to ensure uniqueness.

2. Installing external-dns with TXT Encryption

We'll use Bitnami's Helm chart as it offers more configuration options and simplifies AWS Route53 setup compared to the kubernetes-sigs chart:

helm install external-dns \
  --set provider=aws \
  --set aws.zoneType=public \
  --set aws.credentials.accessKey="$AWS_ACCESS_KEY_ID" \
  --set aws.credentials.secretKey="$AWS_SECRET_ACCESS_KEY" \
  --set txtOwnerId=example-owner-id-123 \
  --set "domainFilters[0]=example-tutorial.com" \
  --set policy=sync \
  --set "sources[0]=crd" \
  --set crd.create=true \
  --set crd.apiversion=externaldns.k8s.io/v1alpha1 \
  --set crd.kind=DNSEndpoint \
  --set txtEncrypt.enabled=true \
  --set txtEncrypt.aesKey="" \
  oci://registry-1.docker.io/bitnamicharts/external-dns

Installation Options Explained

Option	Description
`provider=aws`	Specifies AWS Route53 as the provider
`aws.zoneType=public`	Specifies the use of a public zone
`aws.credentials.accessKey`	AWS access key
`aws.credentials.secretKey`	AWS secret key
`txtOwnerId`	TXT record owner ID (can be any value)
`domainFilters[0]`	Domain to monitor
`policy=sync`	DNS record synchronization policy
`sources[0]=crd`	Enables CRD usage
`crd.create=true`	Creates the CRD
`crd.apiversion`	CRD API version
`crd.kind`	CRD kind
`txtEncrypt.enabled=true`	Enables TXT record encryption
`txtEncrypt.aesKey`	AES key for encryption (auto-generated if empty)

After installation, verify that external-dns is running with encryption enabled:

kubectl logs -l app.kubernetes.io/name=external-dns -f

You should see logs indicating encrypted TXT records are enabled:

time="2024-11-06T07:03:47Z" level=info msg="config: {...TXTEncryptEnabled:true...}"
time="2024-11-06T07:03:47Z" level=info msg="Instantiating new Kubernetes client"
time="2024-11-06T07:03:47Z" level=info msg="Using inCluster-config based on serviceaccount-token"
time="2024-11-06T07:03:47Z" level=info msg="Created Kubernetes client https://10.43.0.1:443"
time="2024-11-06T07:03:49Z" level=info msg="Applying provider record filter for domains: [example-tutorial.com. .example-tutorial.com.]"

Verify that the AES encryption key was generated and stored in the secret:

kubectl get secret external-dns -o yaml

You should see the txt_aes_encryption_key field in the secret data:

apiVersion: v1
kind: Secret
data:
  txt_aes_encryption_key: Q2NCSUF6c2I1N215SGY4RWZtWmZvWm1keUl2SHBsTDY=  # Base64 encoded AES key
# ... other fields omitted for brevity

3. Creating DNS Records

Let's verify that external-dns creates encrypted DNS records. We'll use the DNSEndpoint CRD as it requires no actual resources:

# test.example-tutorial.com.yaml
apiVersion: externaldns.k8s.io/v1alpha1
kind: DNSEndpoint
metadata:
  name: test.example-tutorial.com
spec:
  endpoints:
  - dnsName: test.example-tutorial.com
    recordTTL: 180
    recordType: A
    targets:
    - 127.0.0.1

Apply the manifest:

kubectl apply -f test.example-tutorial.com.yaml

After a moment, external-dns will create the DNS records. Check the logs:

time="2024-11-06T07:12:52Z" level=info msg="Desired change: CREATE a-test.example-tutorial.com TXT" profile=default zoneID=/hostedzone/Z00418233KGBJI8AZJFPR zoneName=example-tutorial.com.
time="2024-11-06T07:12:52Z" level=info msg="Desired change: CREATE test.example-tutorial.com A" profile=default zoneID=/hostedzone/Z00418233KGBJI8AZJFPR zoneName=example-tutorial.com.
time="2024-11-06T07:12:52Z" level=info msg="2 record(s) were successfully updated" profile=default zoneID=/hostedzone/Z00418233KGBJI8AZJFPR zoneName=example-tutorial.com.

Verify the DNS records using AWS CLI:

aws route53 list-resource-record-sets --hosted-zone-id Z00418233KGBJI8AZJFPR

You should see both the A record and the encrypted TXT record:

{
    "ResourceRecordSets": [
        {
            "Name": "test.example-tutorial.com.",
            "Type": "A",
            "TTL": 180,
            "ResourceRecords": [
                {
                    "Value": "127.0.0.1"
                }
            ]
        },
        {
            "Name": "a-test.example-tutorial.com.",
            "Type": "TXT",
            "TTL": 300,
            "ResourceRecords": [
                {
                    "Value": "\"YwPTDxmRgtKjryuSqYrqA35DoRkFw94ZxoojvZ9goHiyXbd8zYS8wBqS7t3ZtZoqREqDDaLtLcB0wbzTpw9n1+HxgGrJc795b4ISnJXRI03+sJ+DgN71dU7hCCyoPx25w/jYbOX3/zP DP59BmZaAly/OLmCEcDTW7dl697qdj4lsNHBrr+6Z1lAFKHAKfX3pM9w6RFGmpGl4WULtAA==\""
                }
            ]
        }
    ]
}

With TXT encryption enabled (txtEncrypt.enabled=true), the TXT record content is encrypted using AES encryption. While it still contains the same management information (heritage, owner, and resource), it's now secured from unauthorized access.

Cleanup

1. Removing DNS Records

Delete the CRD resource:

kubectl delete -f test.example-tutorial.com.yaml

external-dns will remove the DNS records shortly after. Check the logs:

time="2024-11-06T05:48:24Z" level=info msg="Desired change:  DELETE  a-test.example-tutorial.com TXT" profile=default zoneID=/hostedzone/Z08033563HFN15GSXJ766 zoneName=example-tutorial.com.
time="2024-11-06T05:48:24Z" level=info msg="Desired change:  DELETE  test.example-tutorial.com A" profile=default zoneID=/hostedzone/Z08033563HFN15GSXJ766 zoneName=example-tutorial.com.
time="2024-11-06T05:48:24Z" level=info msg="Desired change:  DELETE  test.example-tutorial.com TXT" profile=default zoneID=/hostedzone/Z08033563HFN15GSXJ766 zoneName=example-tutorial.com.
time="2024-11-06T05:48:24Z" level=info msg="3 record(s) were successfully updated" profile=default zoneID=/hostedzone/Z08033563HFN15GSXJ766 zoneName=example-tutorial.com.

2. Removing external-dns

Delete the Helm release:

helm uninstall external-dns

3. Removing the Hosted Zone

First, list the zones:

aws route53 list-hosted-zones

Then delete the zone using its ID:

aws route53 delete-hosted-zone --id Z00418233KGBJI8AZJFPR

Conclusion

In this article, we've explored how to secure external-dns management information by implementing TXT record encryption. While TXT Registry offers a vendor-independent way to store management information, encryption adds an essential security layer. By following these steps, you can maintain the benefits of TXT Registry while ensuring your management information remains secure.

When combined with the setup described in the previous article, you'll have a robust and secure DNS automation solution. Give it a try in your environment!

Forem: suin

Kubernetes: Should You Name Your Controller "foo-bar" or "foobar"? A Survey of 13 Open-Source Projects

What you'll learn from this post

Methodology and scope

Master comparison table

Project-by-project breakdown

1. cert-manager

2. Argo CD

3. Istio

4. Crossplane

5. Knative

6. Strimzi Kafka Operator

7. Prometheus Operator

8. Velero

9. Tekton Pipelines

10. Flux CD

11. Cluster API (CAPI)

12. KubeVirt

13. Kyverno

Cross-project analysis

Go file and directory names

Controller name strings

Finalizer names

Field manager names

Reconciler struct names

Conclusions

The dominant convention: concatenated lowercase

Who uses hyphenation?

The -controller suffix

Recommended naming convention

Why concatenated lowercase wins

Hyphenation has its merits too

Final thoughts

controller-runtime: What Happens When You Do Partial Server-Side Apply?

What You'll Learn

Purpose of the Experiment

Experiment 1: What Happens When You Omit Fields Managed by the Same Manager?

Step 1: First Apply (breed field only)

Step 2: Second Apply (color field only, breed omitted)

Observation

Why Was the Field Deleted?

Supplementary Experiment: What Happens When Another Field Manager Appears?

Step 3: Third Apply by a Different Manager

Observation

Summary

Flagger Does Not Manage Services with Names Different from Their Deployments

Flagger Does Not Manage Services with Names Different from Their Deployments

Test Environment

Testing Method

gatewayclass.yaml

gateway.yaml

Test Results

Services after deployment

Original Service (podinfo-different-name)

podinfo-primary service created by Flagger

podinfo-canary service created by Flagger

Service with the same name as Deployment (podinfo) created by Flagger

Conclusion

The 'hosts' Field in Flagger Canary Resources Can Be Omitted

TL;DR

Background and Purpose

Environment

Testing Procedure

1. Create a Gateway Resource

2. Create the Initial Deployment

3. Create a Canary Resource Without the hosts Field

4. Check the Canary Resource Status

5. Verify HTTP Requests

6. Update the Version

7. Monitor the Canary Release Progress

8. Verify HTTP Requests After Update

Test Results

Conclusion

References

Troubleshooting Docker credsStore Auto-Configuration Issues in VS Code Dev Containers

The Problem

Root Cause

Understanding Docker Credential Helper and VS Code Dev Containers Extension

The REMOTE_CONTAINERS_IPC Environment Variable

Solutions

The `-controller` suffix

1. Remove `credsStore` from the Container's `~/.docker/config.json`

2. Disable `dev.containers.dockerCredentialHelper`

3. Enable `REMOTE_CONTAINERS_IPC` for SSH/Alternative Shells