Forem: Patrick Domnick

Packaging and Uploading a Docker Container and Helm Chart to GitLab using GitLab CI, Operator Framework, Kustomize, and Helmify

Patrick Domnick — Sun, 25 Feb 2024 17:04:44 +0000

This guide serves as a direct follow-up to Building a Kubernetes Operator with the Operator Framework.
It meticulously walks you through the intricacies of packaging a Docker container and Helm Chart, subsequently uploading them to GitLab using GitLab CI, with a keen emphasis on generating semantic version tags. It's noteworthy that nearly all CI templates can be executed locally with minor adjustments to environment variables and utilizing Docker.

Prerequisites

Before commencing, ensure that you have the following tools installed:

yq: brew install yq
docker: brew install docker
kustomize: brew install kustomize
helm: brew install helm
helmify: brew install arttor/tap/helmify

Release Strategy

Before delving into packaging the Docker Container and Helm Chart, let's discuss the strategy in detail. The aim is to create Semantic Releases on a Pipeline Schedule (weekly) to eliminate the need for manual releases. These releases will trigger a pipeline running on a tag, responsible for releasing the Helm Chart and Docker Image. Since pushing changes directly to production is not an option, a pipeline for testing new features and bug fixes will also be implemented.

The workflow unfolds as follows:

Create a new feature branch (feature/improvement).
Make changes to the code.
Submit a Merge Request to the Main Branch (feat: Improve Code). This triggers a pipeline where a Pre-Release is built.
Merge the changes to the Main branch after thorough testing.
Wait for the weekly release to increment the version number (specified as feat, resulting in a Minor Release).
The new tag will publish a Docker Image and Helm Chart to GitLab.

Build the Docker Image

The Operator SDK conveniently provides a Dockerfile, simplifying the building process. We will utilize Kaniko for its user-friendly approach and the fact that it doesn't require a Docker Socket or any form of Docker in Docker. This process ensures the Docker Image is uploaded to the GitLab Container Registry.

.docker:
  stage: build
  image:
    name: gcr.io/kaniko-project/executor:v1.20.1-debug
    entrypoint: [""]
  script:
    - echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"}}}" > /kaniko/.docker/config.json # Authenticate against Gitlab
    - /kaniko/executor --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/Dockerfile  --destination $CI_REGISTRY_IMAGE:$TAG # Build and Push the Docker Image

Helm

Before delving into the process, it's essential to decide where to store Helm Charts within GitLab. Two viable options are:

Package Registry
GitLab Pages

While the Package Registry appears ideal, it comes with a drawback - a somewhat unconventional domain and Helm Chart path (https://gitlab.com/api/v4/projects/{projectId}/packages/helm/stable). Alternatively, GitLab Pages introduces its own set of challenges. Since Pages primarily deals with static code, properly indexing every version becomes nearly impossible. This guide, however, will focus on indexing the latest version, providing a glimpse into this method.

Packaging the Helm Chart

The Operator SDK employs Kustomize instead of Helm, necessitating additional steps to arrive at our Helm Chart. The initial step involves generating the bundle containing our manifests.

.bundle:
  stage: build
  image: 
    name: quay.io/operator-framework/operator-sdk:v1.32
    entrypoint: [""]
  script:
    - make bundle # Generate the Bundle directory dynamically
  artifacts:
    paths:
      - bundle

Ensure to run the make bundle command at least once locally. This ensures the creation of necessary files for automation later on. Additionally, add the bundle/* directory to the .gitignore file. Also, remember to update the Docker Image of your Manager to the actual Docker Image (registry.gitlab.com/path/to/operator).

Generate and Upload the Helm Chart

Now that we have the necessary manifests, it's time to transform them into a proper Helm Chart. For this purpose, we will utilize Helmify and a custom Docker Image bundling useful tools for Helm Chart upload. After generating the Helm Chart, YQ will be used to set the version. As noted in the release strategy, there is no distinction between Chart and Docker Version. In other words, the Application Version and Helm Chart Version will always align. In this example, we will publish the Helm Chart to both GitLab Pages and the Package Registry.

.upload:
  stage: helm
  variables:
    HELM_EXPERIMENTAL_OCI: 1
  image: registry.gitlab.com/stammkneipe.dev/operator-packager:latest
  before_script:
    - git config --global --add safe.directory '*'
    - VERSION=$(git describe --tags `git rev-list --tags --max-count=1` 2>/dev/null)
    - if [ -z "$VERSION" ]; then VERSION="0.1.0"; fi
    - if [ -z "$CI_COMMIT_TAG" ]; then VERSION="${VERSION}-${CI_PIPELINE_IID}"; fi
  script:
    - kustomize build config/default | helmify ${CI_PROJECT_NAME} # Generate the Helm Chart with Kustomize and Helmify
    - yq e -P ".version = \"${VERSION}\"" -i ${CI_PROJECT_NAME}/Chart.yaml # Set the Version of the Helm Chart
    - yq e -P ".appVersion = \"${VERSION}\"" -i ${CI_PROJECT_NAME}/Chart.yaml # Set the Version of the Application
    - helm package ${CI_PROJECT_NAME} --destination ./public
    - helm repo add --username gitlab-ci-token --password ${CI_JOB_TOKEN} ${CI_PROJECT_NAME} ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/helm/stable # Only for Package Registry
    - helm plugin install https://github.com/chartmuseum/helm-push # Only for Package Registry
    - helm cm-push ./public/${CI_PROJECT_NAME}-${VERSION}.tgz ${CI_PROJECT_NAME} # Only for Package Registry
    - helm repo index --url https://${CI_PROJECT_NAMESPACE}.gitlab.io/${CI_PROJECT_NAME} . # Only for Gitlab Pages
    - mv index.yaml ./public # Only for Gitlab Pages
  artifacts:
    paths:
      - public # Gitlab Pages

Generating the Actual GitLab CI Jobs

Up until now, we've been creating templates for reuse in actual CI Jobs. Now it's time to generate the full [GitLab CI] configuration.

Stages

For this pipeline, we only need three stages. The build stages will handle building the Docker Image and the manifests for our Helm Chart, while the release stage will exclusively manage the Semantic Release.

stages:
  - build
  - helm
  - release

Feature Branches

Feature or Renovate jobs will only run when a Merge Request is made to your Main Branch:

.feature_renovate:
  variables:
    TAG: $CI_COMMIT_REF_SLUG-$CI_PIPELINE_IID
  rules:
    - if: $CI_MERGE_REQUEST_SOURCE_BRANCH_NAME =~ /^renovate/ && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == $CI_DEFAULT_BRANCH
    - if: $CI_MERGE_REQUEST_SOURCE_BRANCH_NAME =~ /^fix/ && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == $CI_DEFAULT_BRANCH
    - if: $CI_MERGE_REQUEST_SOURCE_BRANCH_NAME =~ /^feat/ && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == $CI_DEFAULT_BRANCH
    - if: $CI_MERGE_REQUEST_SOURCE_BRANCH_NAME =~ /^feature/ && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == $CI_DEFAULT_BRANCH

containerize_feature:
  extends:
    - .docker
    - .feature_renovate

bundle_feature:
  extends:
    - .bundle
    - .feature_renovate

upload_feature:
  extends:
    - .upload
    - .feature_renovate

Main Branch

The Main Branch will serve two purposes. Firstly, we want to create a latest build, and on top of that, we want to generate semantic releases. This requires ensuring that semantic releases only run on a schedule, and other jobs do not run on schedule.

.latest:
  variables:
    TAG: latest
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "schedule"

containerize_latest:
  extends:
    - .docker
    - .latest

bundle_latest:
  extends:
    - .bundle
    - .latest

upload_latest:
  extends:
    - .upload
    - .latest

release_weekly:
  stage: release
  image: registry.gitlab.com/stammkneipe.dev/semantic-release:latest
  script:
    - git config --global --add safe.directory '*'
    - npx -p @semantic-release/changelog -p @semantic-release/exec -p @semantic-release/git -p @semantic-release/gitlab semantic-release
  rules:
    - if: $CI_PIPELINE_SOURCE == "schedule"

Version Tag

The last jobs will be used for the actual versioned release and only run on a tag.

.tag:
  variables:
    FULL_IMAGE_NAME: $CI_COMMIT_TAG
  rules:
    - if: $CI_COMMIT_TAG

containerize_tag:
  extends:
    - .docker
    - .tag

bundle_tag:
  extends:
    - .bundle
    - .tag

upload_tag:
  extends:
    - .upload
    - .tag

Creating the Schedule

The only thing left to do is to create a schedule to your liking. Keep in mind that Semantic Release will not create a release when there was no change in your code base. So there is no harm in using a short interval.

Usage

Now you can choose between your latest version inside Pages or use the Registry Package (https://gitlab.com/api/v4/projects/{projectId}/packages/helm/stable) to install your Helm chart.

Conclusion

By following these steps, you've successfully set up a GitLab CI pipeline for packaging and deploying a Docker container and Helm Chart. The integration of semantic versioning ensures a structured release process for your application. This streamlined workflow enhances collaboration and facilitates the continuous delivery of your containerized applications.

Building a Kubernetes Operator with the Operator Framework

Patrick Domnick — Sun, 07 Jan 2024 15:46:58 +0000

Overview

Kubernetes Operators simplify the management of complex applications on Kubernetes. In this guide, we'll walk through creating a simple Kubernetes Operator using the Operator Framework. We'll also cover setting up a local Kubernetes cluster with KIND (Kubernetes in Docker) and deploying the Operator to the KIND cluster.

Note: This guide assumes you know what Kubernetes and Docker are and that you have a Mac or Linux/WSL machine.

Prerequisites

You might want to install the following tools on your machine:

golang: brew install go
docker: brew install docker
kind: brew install kind
kubectx: brew install kubectx
operator-sdk: brew install operator-sdk
kubebuilder: brew install kubebuilder

Here are some extra tools which might be useful in the future:

k9s: brew install k9s
kustomize: brew install kustomize
helm: brew install helm
helmify: brew install arttor/tap/helmify

Creating a Kubernetes Cluster with KIND

Before we can deploy our Operator, we need a Kubernetes cluster to deploy it to. We'll use KIND to create a local Kubernetes cluster.

Create a KIND cluster:
```
kind create cluster
```
Configure kubectl to use the KIND cluster:
```
kind export kubeconfig
```
Switch to the correct cluster context:
```
kubectx kind-kind
```
Verify the cluster is running:
```
kubectl cluster-info
```

We now have a minimal local Kubernetes cluster running on our machine. You should be able to use kubectl or k9s to interact with the cluster like any other Kubernetes cluster.

Initializing the Operator Project

We can now build our Operator using the Operator Framework. We'll use the Operator SDK to scaffold a new Operator project and then generate Custom Resource APIs. It is recommended to create a new Git repository for your Operator project and choose a meaningful name for your Operator. You can commit your changes after each CLI command to better understand what the operator is generating.

Create a new Operator project:

operator-sdk init --plugins go/v3 --repo github.com/my-group/my-operator

Create a new Custom Resource Definition (CRD):

operator-sdk create api --group=example --version=v1alpha1 --kind=MyApp

This should leave you with a ready-to-use Operator project scaffolded by the Operator SDK. The three most important directories are:

api: Containing the types for your Custom Resources Definition
controllers: Containing the logic for your Operator
config/samples: Containing sample Custom Resource instances

Implementing the Operator

We'll now implement the Operator logic. The Operator Framework provides a high-level API for writing Operators in Go. This operator will watch for instances of the Custom Resource and create a Config Map for each instance. This is just a simple example to help you get started.

Defining the Custom Resource

Open the file api/v1alpha1/myapp_types.go and add the following code of the MyAppSpec struct:

// MyAppSpec defines the desired state of MyApp
type MyAppSpec struct {
    // INSERT ADDITIONAL SPEC FIELDS - desired state of cluster
    // Important: Run "make" to regenerate code after modifying this file

    // Name is the name of the config to be created of MyApp.
    Name string `json:"name,omitempty"`
}

This will add a Name field to the Custom Resource Spec. We'll use this field to set the name of the Config Map.

We can also add a status field to the Custom Resource. This will be used to store the status of the Custom Resource. To do so, add the following code to the MyAppStatus struct:

// MyAppStatus defines the observed state of MyApp
type MyAppStatus struct {
    // INSERT ADDITIONAL STATUS FIELD - define observed state of cluster
    // Important: Run "make" to regenerate code after modifying this file
    Conditions []metav1.Condition `json:"conditions,omitempty" patchStrategy:"merge" patchMergeKey:"type" protobuf:"bytes,1,rep,name=conditions"`
}

To support visual feedback for the users who are using tools like Openlens, we can add a +kubebuilder:printcolumn annotation to the MyApp struct. To do so, add the following code to the MyApp struct:

// MyApp is the Schema for the myapps API
// +kubebuilder:printcolumn:name="Name",type="string",JSONPath=".spec.name",description="The name of the config map to be created"
// +kubebuilder:printcolumn:name="Status",type="string",JSONPath=".status.conditions[?(@.type==\"App\")].reason",description="The status of this resource"
type MyApp struct {
    metav1.TypeMeta   `json:",inline"`
    metav1.ObjectMeta `json:"metadata,omitempty"`

    Spec   MyAppSpec   `json:"spec,omitempty"`
    Status MyAppStatus `json:"status,omitempty"`
}

Implementing the Controller

Before we can implement the Controller, we need to add a dependency. To do so, open the file controllers/myapp_controller.go and add the following code to the imports:

import (
    "context"

    corev1 "k8s.io/api/core/v1"
    apierrors "k8s.io/apimachinery/pkg/api/errors"
    "k8s.io/apimachinery/pkg/api/meta"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/apimachinery/pkg/runtime"
    "k8s.io/apimachinery/pkg/types"
    ctrl "sigs.k8s.io/controller-runtime"
    "sigs.k8s.io/controller-runtime/pkg/client"
    "sigs.k8s.io/controller-runtime/pkg/log"

    examplev1alpha1 "github.com/my-group/my-operator/api/v1alpha1"
)

Open the file controllers/myapp_controller.go and add the following code to the Reconcile function (// TODO(user): your logic here):

// For more details, check Reconcile and its Result here:
// - https://pkg.go.dev/sigs.k8s.io/controller-runtime@v0.14.1/pkg/reconcile
func (r *MyAppReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    _ = log.FromContext(ctx)

    // Get CRD
    myApp := &examplev1alpha1.MyApp{}
    if err := r.Get(ctx, req.NamespacedName, myApp); err != nil {
        if apierrors.IsNotFound(err) {
            log.Log.Info("MyApp not found. Ignoring since object must be deleted.")
            return ctrl.Result{}, nil
        }
        log.Log.Error(err, "Failed to get MyApp.")
    }

    // Start the Reconciliation
    conditions := &myApp.Status.Conditions
    if len(*conditions) == 0 {
        meta.SetStatusCondition(conditions, metav1.Condition{
            Type:    "App",
            Status:  metav1.ConditionUnknown,
            Reason:  "Initializing",
            Message: "Starting reconciliation",
        })
        log.Log.Info("Condition", "Length", len(myApp.Status.Conditions))
        if err := r.Status().Update(ctx, myApp); err != nil {
            log.Log.Error(err, "Failed to update MyApp status")
            return ctrl.Result{}, err
        }
        // Start the next cycle
        return ctrl.Result{}, nil
    }

    // Act depending on the Condition. This is just a rough example.
    currentCondition := (*conditions)[0].Reason
    switch currentCondition {
    case "Initializing":
        // Create a ConfigMap
        cm := &corev1.ConfigMap{}
        err := r.Get(ctx, types.NamespacedName{Name: myApp.Name,    Namespace: myApp.Namespace}, cm)
        if err != nil {
            if apierrors.IsNotFound(err) {
                // No ConfigMap exists and we create one
                cmInstance := &corev1.ConfigMap{
                    ObjectMeta: metav1.ObjectMeta{
                        Name:      myApp.Name,
                        Namespace: myApp.Namespace,
                    },
                }
                if err := r.Create(ctx, cmInstance); err != nil {
                    log.Log.Error(err, "Failed to create a new  ConfigMap", "Namespace", myApp.Namespace,    "Name",    myApp.Name)
                    return ctrl.Result{}, err
                }
            } else {
                // Some unknown error occurred
                log.Log.Error(err, "Failed to get ConfigMap",   "Namespace", myApp.Namespace, "Name", myApp.Name)
                return ctrl.Result{}, err
            }
        }
        // Update the status
        log.Log.Info("Config Map created")
        meta.SetStatusCondition(&myApp.Status.Conditions, metav1    Condition{
            Type:    "App",
            Status:  metav1.ConditionTrue,
            Reason:  "Available",
            Message: "Config Map created",
        })
        if err := r.Status().Update(ctx, myApp); err != nil {
            log.Log.Error(err, "Failed to update status")
            return ctrl.Result{}, err
        }
    case "Unavailable":
        // Retry depending on the error
    case "Available":
        // Everything is fine
    default:
        // Set State if State was unknown
        meta.SetStatusCondition(conditions, metav1.Condition{
            Type:    "App",
            Status:  metav1.ConditionUnknown,
            Reason:  "Initializing",
            Message: "Starting reconciliation",
        })
        if err := r.Status().Update(ctx, myApp); err != nil {
            log.Log.Error(err, "Failed to update myApp status")
            return ctrl.Result{}, err
        }
    }

    return ctrl.Result{}, nil
}

The Operator will watch the Custom Resources and react to changes. The Reconcile function will be called for each change. The Reconcile function will check the status of the Custom Resource and act accordingly. In this example, we'll update the status of the Custom Resource and create a Config Map if it doesn't exist.

Creating a sample Custom Resource Instance

To test our implementation we'll create a sample Custom Resource instance. Open the file config/samples/example_v1alpha1_myapp.yaml and add the following code:

apiVersion: example.stammkneipe.dev/v1alpha1
kind: MyApp
metadata:
  labels:
    app.kubernetes.io/name: myapp
    app.kubernetes.io/instance: myapp-sample
    app.kubernetes.io/part-of: my-operator
    app.kubernetes.io/managed-by: kustomize
    app.kubernetes.io/created-by: my-operator
  name: myapp-sample
spec:
  name: myapp-config-map

Testing the Operator locally

Deploy the CRDs to the cluster:
```
make install
```
Deploy the sample Custom Resource instance to the cluster:
```
kubectl apply -f config/samples
```
Start the Operator:
```
make run
```

You should now see the Operator's logs in your terminal. The final message should be Config Map created. You can now stop the operator and check your cluster for the Config Map:

kubectl get configMap myapp-sample -o yaml

Conclusion

Congratulations! You've successfully built a Kubernetes Operator using the Operator Framework on a local KIND cluster. You can extend this example by adding more features to your Operator and exploring advanced Operator Framework capabilities. I encourage you to play around with the Operator Framework and explore the possibilities of Operators on Kubernetes.

As you might have noticed, this Tutorial does not include any tests, packaging, or deploying the Operator to a real Kubernetes cluster. I'll cover these topics in future guides. So stay tuned!

Using GitLab Kubernetes Runners to Build Melange Packages

Patrick Domnick — Thu, 28 Dec 2023 10:06:07 +0000

Motivation

Recently, I came across Chainguard and wrote the article How to build Docker Images with Melange and Apko. As a fervent supporter of Kubernetes and GitLab CI, I was eager to experiment with building images using Melange in this particular setup. GitLab's shared Runners work seamlessly with Bubblewrap, eliminating the need for additional configurations. This post is intended for enthusiasts like myself, interested in hosting their own Kubernetes Runners and leveraging the Kubernetes Runner Type of Melange.

Understanding the Kubernetes Executor of Melange

Melange offers four Runner Types:

Bubblewrap (default)
Docker
Lima
Kubernetes

Each has its own security implications and drawbacks. I prefer Docker/Lima for local builds and Kubernetes in a CI environment running on a Kubernetes Cluster. Let's explore how to set this up.

When invoking Melange with the --runner Kubernetes flag, Melange will:

Check if you have sufficient permissions to create pods in Kubernetes (RBAC).
Build a temporary Docker Image and push it to ttl.sh.
Run this Docker Image as a Pod in Kubernetes and build the package.

Remember, this is a high-level overview to understand the necessary configurations and not an explanation of the actual Kubernetes Runner.

Configuring the GitLab Runner

Now that we understand how the Melange Runner works, let's configure our GitLab Runner accordingly. Melange doesn't work with the out-of-the-box Role-Based Access Control (RBAC) Policy of the GitLab Runner. When using GitLab CI with Kubernetes, a Pod (from a Deployment) manages creating Pods for each Job in your GitLab CI Pipeline. This Service Account has the permissions to interact with pods, precisely what we need for Melange. However, Jobs started by this Orchestrator Deployment will only have the default ServiceAccount attached to it. This is fine for normal CI work but not for Melange, as it needs to create new pods and interact with them.

You can change this by modifying the config.template.toml Configmap or the runners.config in the values.yaml. Simply set the service_account for your Kubernetes Runner to the ServiceAccount of the Orchestrator. Here's an example if you named your Helm Release and thus the ServiceAccount gitlab-runner:

runners:
  config: |
    [[runners]]
      [runners.kubernetes]
        namespace = "{{.Release.Namespace}}"
        service_account = "gitlab-runner"
        image = "alpine"

Remember: This has significant security implications. Now, every Job could start background Pods in your Cluster. While this might seem like a significant risk, anyone with access to your repository could do this anyway.

A small extra note: This is the lazy way to do it without the need to deploy any new ServiceAccount and other RBAC Resources. We just need to tweak one little line in our values.yaml file, which I find convenient. If you find a better and more secure way, please let me know and leave a comment!

Configuring Melange

When using the Kubernetes Runner, Melange looks for a config file named .melange.k8s.yaml. Here you can specify the Namespace of the pod, the container Registry to be used, and other Kubernetes Specs. By default, Melange will attempt to start a pod in the default Namespace. However, since no sane person uses the default namespace, we can easily change it to the Namespace of our GitLab Runner.

While we're at it, we can also change the Registry and Repository to the GitLab Registry instead of using ttl.sh. We don't want everyone to have access to our super secure project, do we?

Let's assume our GitLab Runner is running in a namespace called stammkneipe-dev, and our project has the following path stammkneipe.dev/golang-apko-example:

namespace: stammkneipe-dev
repo: registry.gitlab.com/stammkneipe.dev/gitlab-runner-operator

Configuring GitLab CI

Now we are finally ready to use Melange with the Kubernetes Executor in GitLab CI. The only thing left to do is to ensure that Melange has the permission to push images to the GitLab Registry, since we changed the default from ttl.sh to registry.gitlab.com. GitLab has some predefined Variables that can be used to authenticate against our Registry:

CI_REGISTRY
CI_REGISTRY_USER
CI_REGISTRY_PASSWORD

Our final build job would look something like this:

package:
  stage: build
  image:
    name: cgr.dev/chainguard/melange:latest
    entrypoint: [""]
  before_script:
    - cat ${MELANGE_RSA} > melange.rsa # Assuming you have set a Melange RSA Key as a GitLab CI Variable
  script:
    - 'mkdir ~/.docker && echo -n "{\"auths\": {\"${CI_REGISTRY}\": {\"auth\": \"$(echo -n ${CI_REGISTRY_USER}:${CI_REGISTRY_PASSWORD} | base64)\"}}}" > ~/.docker/config.json'
    - melange build --workspace-dir $CI_PROJECT_DIR --runner kubernetes --signing-key melange.rsa melange.yml
  artifacts:
    paths:
      - ./packages

Conclusion and TL:DR

To use the Kubernetes Executor of Melange with GitLab CI and the Registry, you need to do the following three things:

Set the service_account for the runners.config in the [runners.kubernetes] section to the orchestrator Service Account.
Create .melange.k8s.yaml with your namespace set to the GitLab Runner Namespace and the repo set to the GitLab CI Repo (registry.gitlab.com/<CI_PROJECT_PATH>).
Authenticate against the GitLab Registry: 'mkdir ~/.docker && echo -n "{\"auths\": {\"${CI_REGISTRY}\": {\"auth\": \"$(echo -n ${CI_REGISTRY_USER}:${CI_REGISTRY_PASSWORD} | base64)\"}}}" > ~/.docker/config.json'

Now you should be able to use Melange without the need for any external Registries and using your Kubernetes Cluster to build packages.

Building a Go Package with Melange and a Docker Image with Apko

Patrick Domnick — Thu, 21 Dec 2023 12:52:04 +0000

In this tutorial, we will learn how to create a Go package with Melange, build a Docker with Apko image, and showcase examples using GitLab CI.

Prerequisites

Before we begin, make sure you have the following tools installed on your system:

Keyfile Generation

To use Melange and Apko, we need to create a private and public key pair to sign our artifacts:

melange keygen

You should now see a melange.rsa and melange.rsa.pub file in your directory. These files should not be committed and added to the .gitignore file just like the /packages.
This directory will be created ones we run melange to build our Go Application.

Additionally create a sbom directory with a .gitkeep file and add sbom/sbom-*.* and image.tar to the .gitignore file as they will become relevant once we create the Docker image.

Creating a Go Package with Melange

Initialize a Go module:

go mod init gitlab.com/your-username/golang-apko-example

Please replace your-username with your actual GitLab username in the go mod init command.

Create your Go source code files and write your package logic.

Create a melange.yml file in the root of your package directory. This file will define the build configuration for your package:

package:
  name: golang-apko-example
  version: 0.1.0
  epoch: 0
  description: Build a golang application with melange
  copyright:
    - license: MIT
  target-architecture:
    - x86_64

environment:
  contents:
    repositories:
      - https://packages.wolfi.dev/os
    keyring:
      - https://packages.wolfi.dev/os/wolfi-signing.rsa.pub

pipeline:
  - uses: go/build
    with:
      modroot: .
      packages: .
      output: golang-apko-example
  - uses: strip

Customize the name, version and description fields according to your package's requirements. Also, make sure you check the name of the output.

Build your package locally with Melange:

melange build --signing-key melange.rsa --runner docker melange.yml

This will generate an APKINDEX.json, APKINDEX.tar.gz and .apk file in your packages directory.

Building a Docker Image with apko.yml

Create an apko.yml file in the root of your package directory. This file will define the Docker image build configuration:

contents:
  keyring:
    - https://packages.wolfi.dev/os/wolfi-signing.rsa.pub
  repositories:
    - https://packages.wolfi.dev/os
  packages:
    - wolfi-base
    - ca-certificates-bundle
    - golang-apko-example@local

accounts:
  groups:
    - groupname: nonroot
      gid: 65532
  users:
    - username: nonroot
      uid: 65532
      gid: 65532
  run-as: 65532

archs:
  - amd64

cmd: /usr/bin/golang-apko-example

The important line is the golang-apko-example@local package signaling a local package should be used instead of the wolfi packages.

Build the Image with your local repository appended:

apko build --debug --sbom-path ./sbom/ --repository-append $(pwd)/packages --keyring-append=melange.rsa.pub apko.yml golang-apko-example:latest image.tar

This will build a Docker image as an image.tar with our local package and create SBOM files for our image.

To actually use the image locally we have to load and execute it:

docker load --input image.tar
docker run -it golang-apko-example:latest-amd64

Using Gitlab CI

Environment Variables

With Gitlab CI we can automate this process and make it even better.
However, before we can begin we have to save the melange.rsa and melange.rsa.pub as Gitlab CI Environment Files so we can use them. To do this simply go to Settings --> CI/CD --> Variables and Add variable of the Type File our two files:

MELANGE_RSA: File content of melange.rsa
MELANGE_PUB: File content of melange.rsa.pub

You might want to store your certificates in a safer location then Gitlab Environments but lets just stick with this for now.

Gitlab CI Preparation

Now we can create a .gitlab-ci.yml file in the root of your GitLab repository.

At first we should define some general structure:

stages:
  - build
  - containerize

variables:
  APKO_FILE: "apko.yml"
  MELANGE_FILE: "melange.yml"
  FULL_IMAGE_NAME: $CI_REGISTRY_IMAGE:latest

Just like in our CLI we have 2 Steps to complete:

Build the Golang Application
Build the Docker image

Currently we simply create a Docker image without any versioning: The name of our image will be the name of the repository name including the path and using latest as the tag.

Building the Application with Melange

We can use the same command to build the application in Gitlab CI as we did locally. We just have to make sure that our melange.rsa key exists:

build_package:
  stage: build
  image:
    name: cgr.dev/chainguard/melange:latest
    entrypoint: [""]
  before_script:
    - melange version
    - cat ${MELANGE_RSA} > melange.rsa
  script:
    - melange build --signing-key melange.rsa "${MELANGE_FILE}"
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "schedule"
  artifacts:
    paths:
      - ./packages

Since the default entrypoint of melange is melange itself we have to overwrite to entrypoint so that we can use it in Gitlab CI.
After that we make sure that our key exists and can simply build our application.
The only thing left to do is store the packages directory as an artifact which can use in the next step.

Publishing the Docker Image with Apko

We now need to build and publish the Docker Image to the Gitlab Registry. In this step we will need the public key instead of the private key. In addition we can use the predefined Registry Variables to log into the Gitlab Registry and publish the image:

containerize_package:
  stage: containerize
  image: registry.gitlab.com/stammkneipe.dev/apko-ci:latest
  variables:
    CI_BUILDS_DIR: $CI_BUILDS_DIR
  before_script:
    - apko version
    - apko login -u "${CI_REGISTRY_USER}" -p "${CI_REGISTRY_PASSWORD}" "${CI_REGISTRY}"
    - cat ${MELANGE_PUB} > melange.rsa.pub
  script:
    - apko publish --debug --sbom-path $CI_PROJECT_DIR/sbom/ --repository-append $(pwd)/packages --keyring-append melange.rsa.pub "${APKO_FILE}" "${FULL_IMAGE_NAME}"
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "schedule"
  artifacts:
    paths:
      - "sbom/*"

This will result in a Docker Image directly instead of a tar file. You should be able to see your docker image in the Gitlab Registry (/container_registry/) with information on how to use it.
In addition we can see the SBOM files in the artifacts of the pipeline.

Conclusion

See the full example here.

Accessing external Secrets in Gitlab CI

Patrick Domnick — Sun, 15 May 2022 17:32:22 +0000

There are times when you want to access your existing credentials from a secure location, like Vault by HashiCorp, AWS or some other big Cloud provider but your current CI Image does not have the necessary SDK installed or means to retrieve them. Maintaining them a second time as Gitlab Variables or accessing them in a previous job should not be done and is considered insecure for many reasons. That is why I created Gin Vals. Simply run Gin Vals as an Service and make a Web Request to the service.

Concept

As the name describes Gin Vals combines to simple GO dependencies to create a slim and easy solution for most providers.
Vals by Variantdev is a tool for managing configuration values and secrets for the major cloud providers and other technologies.
Now we simply need to make it accessible via REST with the Gin Web Framework.

Usage

Because we are using Vals as our configuration and secrets tool, we can simply refer to its documentation:

Vault: ref+vault://PATH/TO/KVBACKEND[?address=VAULT_ADDR:PORT&token_file=PATH/TO/FILE&token_env=VAULT_TOKEN]#/fieldkey
AWS Secrets Manager: ref+awssecrets://PATH/TO/SECRET[?region=REGION&version_stage=STAGE&version_id=ID]
GCP Secrets Manager: ref+gcpsecrets://PROJECT/SECRET[?version=VERSION]
and many more...

I will use the simple echo method to display some possible methods of using Gin Vals.

Simple single value GET request

The easiest way is to just retrieve one secret via GET. Add the Vals string as path and you should be able to get your secret value into a variable.

test:
  image: bitnami/bitnami-shell:latest
  services:
    - name: patrickdomnick/gin-vals:latest
      alias: ginvals
  script:
    - export secret=$(curl -X GET "http://ginvals:9090/ref+echo://foo/bar")
    - echo secret

Simple single value POST request

The more advanced method would be to retrieve many secrets at once as a json object. From here we could parse the data with tools like jq depending on the main image you are using.

test:
  image: bitnami/bitnami-shell:latest
  services:
    - name: patrickdomnick/gin-vals:latest
      alias: ginvals
  script:
    - export secretJson=$(curl -H 'Content-Type: application/json' -d '{"foo": "ref+echo://foo/bar","bar": "ref+echo://bar/foo"}' -X POST "http://ginvals:9090")