Forem: Simon Emms

Zigflow: The Missing Temporal DSL

Simon Emms — Mon, 02 Feb 2026 19:33:11 +0000

TL;DR: Zigflow lets you write Temporal workflows in YAML, so you can focus on what happens instead of how to make it reliable.

Temporal is one of those tools that feels like a superpower once it clicks. Durable workflows, automatic retries and crash recovery baked in.

But getting there can feel heavy.

Since I joined Temporal last April, I've really seen first-hand how it helps engineers make their application bullet-proof whilst writing less code.

Last summer, I had a week of customer calls where a common pattern kept repeating. The whole week, I heard some version of the same thing:

Temporal is great for developers, but we want our business users to be able to define workflows. Is there a DSL we can use?

They loved Temporal for its durability and reliability, but they didn't want every workflow change to require an engineer, a code review and a deployment. For them, the workflow wasn't the complexity, but the implementation was.

That tension was the spark for Zigflow.

Temporal is powerful, but there is a learning curve

Temporal workflows are just code, but not necessarily normal code. Workflows are replayed. You have to understand determinism (and non-determinism). Things that once felt natural have to be reconsidered.

For experienced Temporal users, that's fine. It can even feel elegant.
For newbies, it's a whole new paradigm.

Also, most workflows aren't algorithmically complex:

They're sequences.
They branch.
They wait.
They call services.
They loop.
They retry.

Most workflows aren't complex in what they do - they're complex in how they're made reliable.

Enter Zigflow

Zigflow is a declarative DSL for Temporal workflows.

Instead of writing your workflow code directly, you describe your workflow in YAML. Zigflow turns that description into a real Temporal workflow with retries, durability and all the usual Temporal guarantees.

Think of it as a way to start simple with Temporal, without painting yourself in a corner.

You get all the reliability, observability and scalability without having to think about the sharp edges on day one.

A DSL (domain-specific language) is a specialised language, focused on the problem. Rather than inventing yet another DSL, Zigflow builds on the CNCF's Serverless Workflow specification - a vendor-neutral standard designed for exactly this problem.

Show me the money

If you've gotten this far, you'll want to see how it all works.

document:
  dsl: 1.0.0
  namespace: zigflow
  name: hello-world
  version: 1.0.0
do:
  - set:
      output:
        as:
          data: ${ . }
      set:
        message: Hello from Ziggy

This isn't pseudo-code, but a full Temporal workflow ready to receive triggers.

Run this with the Zigflow CLI (zigflow -f ./workflow.yaml) and you've got a running Temporal workflow. Trigger this from the Temporal UI (task queue: zigflow and workflow type: hello-world) and you'll see the response:

{
  "data": {
    "message": "Hello from Ziggy"
  }
}

The goal of Zigflow isn't low-code for the sake of it, but to allow you to focus on what the workflow does. And you get the added bonus of Temporal best practices.

Best practices as guardrails

As a workflow grows, Temporal concepts start to matter:

Histories get long, so you need Continue-As-New
Activities get slow, so you need heartbeats
You want visibility, so you need search attributes

Zigflow bakes all these ideas in from the start, so you don't need to learn them upfront. You still benefit from them, but they're implemented as sensible defaults so you only need to know about them when they matter.

And if you know Temporal already, Zigflow becomes a way to standardise and accelerate workflow creation.

Written in Go. Not locked to Go

Zigflow is written in Go, but the workflows it runs are just Temporal workflows.

That means they're language-agnostic by nature.

You can trigger them from TypeScript, signal from the Temporal UI, query from Python and update from Java.

Why you might want to try it

You're curious about Temporal, but put off by the learning curve
You want to prototype workflows fast
You like the idea of workflows being readable by humans
You're building tooling or platforms on top of Temporal

If any of this resonates, Zigflow is worth a look:

👉 Docs: zigflow.dev
👉 Code: github.com/mrsimonemms/zigflow

If you try it and like it, please add a GitHub star. I know it's a bit of a vanity-metric, but it's great seeing feedback from people who are using it. And that motivates me to keep building.

This is just the beginning and I've got so many ideas about new features (including a drag-and-drop UI). If this resonates, try it out. And I'm planning a follow-up post diving into these ideas.

And let me know what you build.

When "The Best" isn't good enough

Simon Emms — Fri, 14 Jun 2024 00:00:00 +0000

I am a beekeeper.

This won't be of much surprise to anyone who's spent any time with me, or who follows @TheShroppieBeek on theInformation Superhighway. I'll bore for England on the subject of beekeeping. One thing beekeepers often say to each other is:

Ask a beekeeper a question and get two answers

A question that might seem simple, such as "what's the best way of raising a new queen?", will come with a multitude of opinions, folklore and experience. You might ask a fan of the Miller Method, or a lover of Grafting or someone who likes using any of the other methods.

The problem here is asking for "The Best". How do we know it's "The Best"? When I say "The Best", I'm looking for the easiest way of doing it. When you hear "The Best", you might think I'm looking for the most reliable way of doing it. These are subtly different things.

My setup will be different to the person I'm asking:

I have six hives over two apiaries
I'm a hobbyist
it's not my main income, so I don't need to turn a profit
my bees are all fairly sheltered from the wind
my bees all have south or east-facing entrances
my bees are all around ~130 metres above sea-level
my bees are all around 52°N
I select my bees for calmness rather than productivity
I use wooden, National boxes

The person I ask almost certainly won't keep bees in an identical fashion to me. And, even if they did, they will have a natural odour, style and demeanour which the bees will pick up on and react differently to. So if I ask for "The Best", I'm assuming that they will know all about my bees and that there's will be the same.

Which is why you end up getting two answers from one beekeeper.

"The Best" in software engineering

We see a similar behaviour in software engineering. I have been regularly asked for "The Best" without giving any other answers.

Let's examine the question "what's The Best cloud provider?", which is a question I'm asked with depressing regularity. Defining "The Best" is actually quite difficult:

do you need a truly global application (including Africa and South America), or do you just need Europe and North America?
are you using a VPN, or will everything be accessed over the public internet?
are you going to be having vast quantities of traffic/data, or is it only going to be a few gigabytes per month?
do you need to use Windows machines, or is everything on Linux?
do you need managed Kubernetes, or are you comfortable using K3s?
do you have any specialist requirements (like a satellite), or are you just deploying some containers and storing data?
is money no object, or do you need to watch the pennies?

These are just some of the questions that need to be answered in order to give an answer to the question. In my mind, I divide up the cloud providers into "The Establishment" (AWS, GCP, Azure) and "The Challengers" (DigitalOcean, Civo, Hetzner and others). For (most of) the questions I asked, the first part is something that The Establishment caters for (and does well), but the second part is something that everyone does well.

Tell me the things that matter to you and I'll be able to give an intelligent answer.

Define your parameters

I've regularly run post-mortems in my capacity as a technical leader. I often end them by talking of better questions that we can ask in future to avoid the situation that's gone wrong. Telling me how you just "The Best" is useful in helping me understand the question you're actually asking.

Instead of asking for "The Best", trying asking a better question:

my goal is X. What are the different ways I could achieve this?
am I on the right tracks with this? Are there any things I need to watch out for?
I want to avoid X - how might you do it?
I'm doing a proof of concept that I want to move into production - what should I watch out for?
I want to put this application on the internet - how are we doing it for other things like this?

The exception that proves the rule

Of course, there are always exceptions. If you ask aneager-to-please comedian to buy The Taskmaster "the best present", you might just get them to get a tattoo whilst filming an unbroadcast show on a channel known for just showing repeats.

And comedy gold ensues.

NB. If you're Little Alex Hornereading this, please keep asking comedians/intelligent idiots for "The Best". The vagueness is what actually means makes for varied, interesting and funny TV.

Self-Hosted is dead - long live Self-Hosted

Simon Emms — Sun, 05 Feb 2023 00:00:00 +0000

Gitpod may have deprecated support for self-hosted, but that's no reason to let it die. I've created a Gitpod Self-Hosted community installation repository to fly the flag for those of us who want and need self-hosted.

Regular readers will know that I worked at Gitpod untila couple of weeks ago. Whilst I was there, I provided the technical leadership on the self-hosted offering. I planned and built the Gitpod Installer to remove the complexity of our old Helm charts, worked withReplicated to package it for public consumption and various other odds and sods.

In December, Gitpod pulled official support for self-hosted. For anyone who worked alongside me at Gitpod or worked with me in the Gitpod community, you will be aware that I had reservations about this as a strategy. I'm not going to get into the detail of that here (I may do so publicly in the future), but there's a couple of highlights that are appropriate to call out:

Gitpod SaaS works well - this is the shop-window and how the majority of people will interact with it
A managed Gitpod service is a good idea (and something I argued in favour of over a year ago)
Dropping self-hosted excludes a whole host of hobbyists and champions, who were crucial to getting Gitpod to where it is today
It removes Gitpod as a viable option for businesses who need to guarantee things like data sovereignty, access policies or other compliance policies (Dedicated does this, but there will still be those for whom requiring the guarantees will be cost/time-prohibitive, so will insist on on-premise or nothing)
Dedicated will be significantly more expensive than Self-Hosted

This Town Ain't Big Enough For The Both Of Us

Gitpod is difficult to run. It is a Kubernetes application that builds it's own images. It provides access to the Docker socket so that Docker can be used inside it. In order to do this and maintain security and isolation between workspaces, lots of work has been done to make this happen.

This means you can't just get any old Kubernetes instance and run it. Anyone who's thought "oooh, I'll spin up minikube and have a play with Gitpod for half an hour" will know that it doesn't work like that.

Gitpod.io runs on k3s. It used to run on Google's GKE, but it was found to have certain limitations that meant that it wouldn't run reliably on it. In self-hosted though, we supported GKE. And EKS. AndAKS. Strangely, we didn't officially support k3s (although I had created a k3s guidewhich was quite popular in the community). But that means we were officially supporting four times the number of distributions, including one that we couldn't get to work reliably for our own paid service.

Gitpod also needs nodes that run on Ubuntu 20.04. And the nodes could have the shiftfs moduleenabled, or use FUSE if it wasn't. And then there was in-cluster or external database/registry/storage.

And that's not counting the number of times where a feature was added that would have added more requirements on the nodes, such as the time a PR was opened that changed all the persistent volume claims to use a feature that was only available on Google Cloud Platform.

This meant that we were supporting a matrix of many permutations. For a team of four engineers, that was a big task.

It also meant that Self-Hosted was actually slowing down Gitpod's progression. A good example of this is gitpod-io/gitpod#14005. This is a pull request to drop support for FUSE. Alejandro (rightly) said that "Ubuntu stable already provides shiftfs OOTB", but what that didn't take it account of was that the managed Kubernetes nodes didn't necessarily have it enabled. In both GKE and AKS, shiftfs wasn't actually enabled by default. So the ticket lay fallow for months, with no one really happy with this state of affairs.

This had a practical benefit to Gitpod. By only supporting shiftfs, the complexity of the installation would reduce, the image build speeds would dramatically increase and everyone would be happy.

Except those customers of GKE and AKS...

Bright Idea

There are two broad schools of thought for on-premise applications:

provide support for a wide-range of features, potentially limiting the deployment options
provide support for a wide-range of deployment options, potentially limiting the features

Historically, Gitpod fell into the first camp as it was felt that by providing lots of deployment options, we'd get the most customers. Towards the end of 2022, it became clear that this was actually hurting us.

For much of my last 6 months at Gitpod, I was advocating an approach where we limit our deployment options. Instead of supporting the managed Kubernetes of all the major cloud providers, we should only support specific versions of k3s. Our job in the Self-Hosted team would then be:

maintaining the known images of Ubuntu
maintaining ways of autoscaling the VMs in those cloud providers
being able to provide well-tested and reliable instructions to our users

A few weeks before I left, I spoke to one of my teammates and they said:

If you think you're so clever, why not build a proof-of-concept?

So I did.

And, just because I'm no longer a Gitpodder, I'm still a lover of Gitpod so I continued working on it after I left (that's normal right?).

At this stage, it only has support for Hetzner as this is what I use (it's cheap as chips, fast and reliable), but there's plans to add more cloud providers in future. Importantly, we have control of the configuration of the Kubernetes runtime and the configuration of the nodes that these run on.

It creates a pool of Kubernetes managers and a pool of Kubernetes nodes. In a typical managed instance, you don't actually have access to the managers (usually known as the control-plane) and you only have access to the nodes. But, because this is k3s, you have access to both. Once Terraform has created the managers, it installs k3s to them.

When you're scaling a cluster, you typically wouldn't be scaling the control-plane because these would not normally have any Gitpod resources on them. So, when you scale, you're adding new nodes.

Both the managers and nodes make use of cloud-initscripts to configure the node for us, which means we don't actually need to do anything clever like creating base images in Packer. For themanager VMs, the cloud-init script installs the required packages, change the SSH port from 22 (Gitpod needs port 22 for it's own SSH access) and installs shiftfs. For thenodes it does exactly the same and then installs k3s and connects it to the manager pool.

The reason that the cloud-init script has the k3s connection information? So that we can add a new node via autoscaling without needing someone to run the Terraform scripts. Importantly, creating a new VM from scratch only takes about a minute or so, which is about the same amount of time that most managed Kubernetes services take to provision a new node.

I Am The Resurrection

I don't know where I'm going with this project in truth. I'm a passionate and empathetic engineer, so I can't just turn off the taps on something that was my entire working life for the past 18 months. I guess if it had ended on my terms, then that's one thing - by being made redundant, I finished mid-ticket and I had the very hollow feeling of unfinished business.

I worked bloody hard to make Gitpod Self-Hosted a success and I hated seeing that it wasn't as successful as it should have been. Gitpod is an open-source project, so this could well become a vibrant, community-remix of Gitpod. Equally, it could just be something I do for myself. I have no influence over the technical direction of Gitpod any more, so I guess this could all be closed down very quickly if they don't want it to exist any more.

It's unlikely that I'll do work on other cloud providers just because. I use Hetzner and this works well for me. If you want me to open up other cloud providers, I will want sponsoring for this effort (and access to an account for the desired cloud) so that I'm not funding development of a commercial, VC-backedcompany that recently decided it couldn't afford me any more. I love Gitpod and it's community, but maybe not that much.

I will be writing some extensive documentation for the project over the next few weeks, so please keep an eye on the project. And, if you want to discuss me working on additional cloud providers, please put a call in my diary so we can discuss things.

Resources

GitHub repository: github.com/mrsimonemms/gitpod-self-hosted
Actual usage: github.com/mrsimonemms/gitpod-app

I am an ex-Podder

Simon Emms — Sun, 29 Jan 2023 15:02:01 +0000

Hey Simon, I'm sorry to be sending this. We've made the hard decision to reduce the size of the Gitpod team and you are affected.

That was the Slack message I woke up to on the morning of Tuesday 24th January 2023 from our CTO and my time with Gitpod was over, along with 20 other colleagues and friends.

So, guess it's time to find something else then.

One of the most recurring questions I've had in the past few days is "what is it you're looking for Simon?". The first thing to say it that, until 10am on Tuesday morning, I wasn't planning on looking for anything. But that ain't going to pay the mortgage, so here's some thoughts...

I have about 15 years of experience and I'm bloody good at my job. I don't want to be Engineer #7,384 at a company. I have been a technical leader at many places and want to go somewhere that would benefit from that experience.

The job title doesn't matter, but what I'm doing does (that said, bonus points if I have a job with a comical acronym). Dependent upon the structure, the ideal job might be CTO, Staff Engineer, Architect or Senior Engineer. The important thing is that I want a role where I can work with other brilliant people and provide the technical leadership to either the whole company or a section of it.

I care about what I'm doing. When I went to Gitpod, I took both a salary cut and a responsibility cut, but I did that because I believed in what I was doing. I want to go somewhere where I'm working on something that matters.

I like working on difficult problems. They're exciting, interesting and challenging.

In 2022, I did four talks in the UK, Spain and the USA (I'm still claiming KubeCon as I wrote the talk, even though I caught Covid the week beforehand so CTO Chris had to deliver the talk). This was something that I really enjoyed and I had very positive feedback about the talks. I don't want
DevRel to be the main focus of my next job, but I would very much like it if it was something that I was supported to do as a sideline.

I want to work with a brilliant and diverse group of people. I've loved working with engineers from across the world at Gitpod, learning about different ways of working, different cultures and different parts of the world. Bonus points if you have a hiring policy that levels the inherent privilege of straight, white, heterosexual male engineers (like myself).

Remote working only. I'm fuelled by tea and I only buy the best (Yorkshire Tea, natch). After the pandemic, this really shouldn't even be a discussion point any more, but I wrote some thoughts on the subject pre-pandemic.
I live in a beautiful part of the country and my desk looks out onto my garden where I have goldfinches, robins, sparrows and blackbirds cavorting around in front of my desk all day long. When I need 10 minutes away from my desk to mull over a problem, I'll go and watch my bees foraging. A quiet, inspiring office with good tea makes me work better - it certainly beats commuting.

Money is NOT the motivating factor. As a guide, I'm looking in the region of £100,000 - the closer the role is to my ideal then the more negiotiable this is. If the role isn't particularly close to my ideal, then the money IS the deciding factor - in that case, to quote Spike Milligan:

All I ask is the chance to prove that money can't buy you happiness

And I'm not going to do a technical test. By all means, I'm happy to prove my credentials in a conversation, but technical tests are a fundamentally broken and flawed system. And asking someone with my experience is a waste of time (and fairly insulting). If you need to see I can code, you can look at my
GitHub or testimonials from the people I've worked with.

Finally, the only absolute no-no. NO GAMBLING COMPANIES. Ever. They're a cancer on our society, preying on the most vulnerable people and who provide little-to-no value except to their shareholders. I don't want to go to sleep at night knowing that my house is being paid for by people suffering with addiction problems.

This is my ideal. I don't expect all of these to be met by my next role, but I want to get close. If you have something you think is suitable, please put a time in my diary so you can tell me about it.

Building a RESTful API With Functions

Simon Emms — Sun, 29 Jan 2023 15:00:33 +0000

There is an accompanying GitHub registry with a working demo

Serverless functions are a great way of building a scalable application. They're small, independent, fast-to-build and infinitely scalable. If you need to build an application with just a couple of endpoints, they will work great.

When it comes to building a big application or one that you want to expose as a public service, serverless functions can end up being just a bit cumbersome to develop. As most serverless frameworks focus on single functions, there can be a lot of repetition in getting a CRUD application set up.

What is a RESTful application?

For more information, check out the Wikipedia article.

REST (Representational state transfer) is a way of representing data between machines. It has a several key features:

no session state is maintained between calls
the HTTP verb decides how the request is handled
it returns the data model

What tech are we using?

Whilst there's plenty of open-source tech out there that can help, this is my favoured way of achieving the end goal:

OpenFaaS for the serverless functions
Kong for the API gateway
Kubernetes and Helm for deployment
K3d and Skaffold for local development
A custom NodeJS OpenFaaS template that uses MongoDB and Mongoose to manage the data models

Getting started

As I like to keep things simple, all you need to do to get the development cluster up and running is to run make. This will run a series of commands to check you have the correct dependencies, create your k3d cluster and provision your cluster. Once you've done that, you can access your cluster on localhost:9999.

Once you've got the cluster up and running, you can just use make serve to reload the Kubernetes objects.

Your first function

In the example repo, look at the product function as the first function

To create your first function, run the command FN_NAME=product make new. This will create a new OpenFaaS function in the /components directory.

The important file is schema.js which is a standard Mongoose schema. In this example, we're just defining a modelName of Product - in a more complex example, we can add in both synchronous and asynchronous validation, but a simple name property will do for now.

module.exports = ({ model }, BaseSchema) => {
  const modelName = 'Product';

  const ProductSchema = new BaseSchema({
    name: {
      type: String,
      required: true,
    },
  });

  return model(modelName, ProductSchema);
};

Next, add this to the functions Helm chart in /chart/functions/values.yaml.

functions:
  product:
    image:
      repository: ghcr.io/mrsimonemms/openfaas-rest-api/product
      tag: latest
    envvars:
      LOGGER_LEVEL: info
      MONGODB_URL: "mongodb://openfaas-mongodb.openfaas.svc.cluster.local:27017/openfaas"

Finally, add the image to the artifactOverrides section in the skaffold.yaml file.

artifactOverrides:
  functions:
    product:
      image: ghcr.io/mrsimonemms/openfaas-rest-api/product

If you visit the OpenFaaS dashboard, you will see the product function has been deployed to OpeenFaaS.

To get the login credentials, run make openfaas

Making it into an endpoint

Now we have an OpenFaaS function running, we need to configure Kong so that it acts as a gateway. OpenFaaS functions are exposed on /function/:name, which is what we're going to redirect to - in this template, the CRUD endpoints live under /crud.

First, in /charts/openfaas/values.yaml, add a product-gateway service:

services:
  - name: product-gateway
    annotations:
      konghq.com/path: /function/product/crud

Finally, in the /charts/openfaas/template/ingress.yaml, tell the Kong ingress about the product-gateway service:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: openfaas
  annotations:
    konghq.com/strip-path: "true"
spec:
  ingressClassName: kong
  rules:
    - http:
        paths:
          - path: /api/v1/product
            pathType: Prefix
            backend:
              service:
                name: product-gateway
                port:
                  name: http

Now you'll find this exposed on localhost:9999/api/v1/product.

Your second function and nested endpoints

In the example, this is the product-size function

So far, we've done a fairly simple example only. Next, we're going to up the complexity by adding a nested endpoint. A root endpoint is fairly straightforward because it just sends everything to the function and returns whatever that returns.

Conversely, a nested endpoint is locked to a product ID. Fortunately, Kong makes this fairly straightforward for us with its plugin system.

Firstly, repeat all the above steps to create a product-size function. In the schema.js, add a productId parameter - this will store the _id from the product function.

This time, when creating the /charts/openfaas/template/ingress.yaml, we're going to add a named path parameter in the path.

- path: /api/v1/product/(?<productId>[\w-]+)/size
  pathType: Prefix
  backend:
    service:
      name: product-size-gateway
      port:
        name: http

This searches for any alphanumeric character between /product/ and /size in the URL and assigns it the name productId.

Next, we have to tell it what to do with it. In the /charts/openfaas/values.yaml, add this to the services array:

- name: product-size-gateway
  annotations:
    konghq.com/path: /function/productsize/crud
    konghq.com/plugins: product-request-transformer

Notice how this refers to a product-request-transformer plugin. So, let's define it in the same values.yaml file.

plugins:
  - name: product-request-transformer
    plugin: request-transformer
    config:
      remove:
        body:
          - productId
      add:
        body:
          - productId:$(uri_captures["productId"])
      append:
        querystring:
          - filter:productId||$eq||$(uri_captures["productId"])

This does a few things. Firstly, it removes any productId from the body and then adds in the product ID in the URL. It also appends a query string filter=productId||$eq||<productId> to the URL. This ensures that the product-size function behaves like a nested API endpoint.

Importantly, this won't return an HTTP 404 result if the product doesn't exist. That's beyond the scope of this demo, although you could achieve this by adding a middleware.js file to the function. This can be either a function or an array of functions and it follows the same basic interface as an Express middleware function.

Conclusion

With this simple demo, you can see that it's very possible to create a fully RESTful API from a collection of serverless functions. It's important to remember that all these functions are entirely isolated from each other from a coding point of view (so you could even do something interesting like writing them in different languages). This makes them very powerful and almost infinitely scalable.

Multi-Arch Docker Containers

Simon Emms — Sun, 29 Jan 2023 14:51:49 +0000

Containers have revolutionised computing since they were first popularised by Docker in the first half of the 2010s. Containers have made
developing software and deploying it far simpler than it has ever been by creating an artifact that can work across different systems. By maintaining it's own dependencies independent of the host machine, we no longer have to ensure that all developers and deployment environments are using version x.y.z of a language and the correct versions of all our databases.

With the exception of multi-architectural deployments that is.

What is multi-arch?

I am currently writing this post on my laptop - it's a 64 bit machine running Ubuntu 19.10. If I run uname -p, it proves that it's a 64 bit machine by printing x86_64. When I come to deploy it to my Raspberry Pi cluster (yes, I'm that cool kids), that'll be on a 32 bit processor - uname -p now gives me armv7l.

This is the description of the processor on the computer itself and it's architecture. When you're running your containers, you get used to largely being able to ignore the host machine and it's capabilities entirely - you might be running Windows or OSX, but if your container is an Ubuntu or Alpine-based image, you'll be doing everything in Linux.

The processor is one of the few things that the container does not virtualise. If you have a 64 bit host machine, your container MUST be compatible with a 64 bit processor. For the most part, this causes us few problems - we all tend to develop on 64 bit machines and deploy to cloud provider of choice who provides us with a fleet of 64 bit machines. This only becomes a problem if we need to support multiple architectures at any stage the of the software development lifecycle.

Why are multi-arch containers a good idea?

Until fairly recently, if you wanted to deploy your containerised application, you were pretty-much limited to doing so to 64 bit machines. It was possible to get Docker Swarm onto a Raspberry Pi and Kubernetes was (officially) off limits.

Then along came Scaleway with their very cheap ARM clouds and K3S with a lightweight Kubernetes that was perfect for Raspberry Pis. Now you can have very cost-effective and (with enough nodes) high-performing clusters running on machines lying around your office. These are perfect for development and staging clusters to test out your applications.

In recent years to, the rise of the Internet of Things (IoT) has largely been made possible by lightweight processors. I've worked with many IoT companies over the years and it's always useful to be able to have a virtual device with which to interact in development and testing - this process is simplified greatly if you have that software in a container.

If none of these reasons convince you based on your requirements today, think about what the future might bring. There have been persistent rumours that Apple will switch to ARM processors in the future (which may or may not require multi-archness). You also rarely know exactly where your application will be heading in 3+ years time - I've lost count of the number of times I've been told by architects and products owners "no Simon, we definitely will never do x feature" only to find I'm building that exact feature 6 months later.

Finally, it's a very simple change that add almost no time or effort to the build pipeline - for the effort involved, is it not worth just having it there in the background?

Docker Setup

Even those the experimental version of Docker is not recommended for production
use, it is fine for building the containers. You do NOT need to enable experimental
mode on your deployment machine. As further evidence for it's use, this is how
Docker provides multi-arch support for all officially supported containers.

In order to make truly multi-arch containers, we need to use the experimental version of Docker. To do that, go to your command line and edit the file ~/.docker/config.json. This is a JSON file, and you need to ensure that experimental is set to enabled.

{
  "experimental": "enabled"
}

It's likely you'll have additional settings in there - leave those as they are. To prove you have enabled experimental mode, type docker manifest --help and you should see the help page.

Next, you need to enable Qemu support for your Docker host instance. Qemu is a generic
machine emulator and virtualiser. In simple terms, it allows your machine with one type of processor to emulate other types of processor, allowing your machine to execute scripts on different processor architectures.

This updates the Docker machine instance. It will only need to be done once for your Docker instance. If you restart your machine or the Docker instance, you will need to run it again.

```shell script
docker run --rm --privileged multiarch/qemu-user-static --reset -p yes




Now you have your Docker instance set up. It's time to consider the Dockerfile. Under normal circumstances, your Dockerfile would start off with using the `FROM` command to pull in an image, perhaps `FROM node:12-alpine`. Now you're in the multi-arch world, you cannot simply do that. Since [2017](https://www.docker.com/blog/docker-official-images-now-multi-platform/) all Docker images are multi-arch - if you just use the image name, eg `node`, it queries the manifest against your host machine's processor and pulls that down. On a 64 bit machine, it actually pulls down `amd64/node`.

To get around that, we need to specify the architecture. Since Docker v17, we can add build arguments before the `FROM` tag. So you will need to change your Dockerfile like so:



```Dockerfile
ARG ARCH="amd64"
FROM ${ARCH}/node:12-alpine
CMD [ "uname", "-a" ]

We're defaulting the ARCH argument to amd64 and then telling Docker to pull down that exact image. If we now build two different images, one for amd64 and one for a Raspberry Pi 3B, we should see differences.

```shell script
docker build -t riggerthegeek/multiarch-test:amd64 .
docker build --build-arg=ARCH=arm32v7 -t riggerthegeek/multiarch-test:arm32v7 .
docker run -it --rm riggerthegeek/multiarch-test:amd64 # String includes x86_64
docker run -it --rm riggerthegeek/multiarch-test:arm32v7 # String include armv7l




Now we have all the images built, we need to combine them into a manifest. As touched on above, a manifest is a way of combining multiple images into a single image, allowing the machine that's pulling the image to decide which one it wants to actually use. There
are many use-cases for this, such as whether the host's operating systems is Windows. In our case, we only want to differentiate by the processor.

> You will need to push the above images before running this command



```shell script
docker manifest create riggerthegeek/multiarch-test \
    riggerthegeek/multiarch-test:amd64 \
    riggerthegeek/multiarch-test:arm32v7
docker manifest push riggerthegeek/multiarch-test

If you run docker manifest inspect riggerthegeek/multiarch-test now, you will see how the host Docker engine will decide which image to use.

Finally, test your container on an AMD64 machine and on a Raspberry Pi. You should see different results, but notice how you're only specifying the image name, not the tag.

```shell script
docker run -it --rm riggerthegeek/multiarch-test # x86_64 on AMD64, armv7l on Raspberry Pi




You have now built a multi-arch Docker image, deployable to both 64 bit machines and Raspberry Pi 3B/3B++.

## Building with GitLab

I use GitLab for my CI. Enabling multi-arch builds in GitLab is really simple - it's one environment variable and then the same commands as above. This is my usual setup in my `.gitlab-ci.yml` - I've removed all branching guards and strategies for brevity.



```yaml
variables:
  DOCKER_CLI_EXPERIMENTAL: enabled # Required for docker manifests - the only change required
  DOCKER_DRIVER: overlay2
  DOCKER_HOST: tcp://docker:2375

stages:
  - build
  - publish

# Extensible commands
.docker_base:
  image: docker:stable
  services:
    - docker:dind
  tags:
    - docker
  before_script:
    - docker login -u ${CI_REGISTRY_USER} -p ${CI_REGISTRY_PASSWORD} ${CI_REGISTRY}
    - docker run --rm --privileged multiarch/qemu-user-static --reset -p yes

.docker_build:
  extends: .docker_base
  stage: build
  script:
    - docker build --build-arg=ARCH=${ARCH} -t ${CI_REGISTRY_IMAGE}/${ARCH}:${CI_COMMIT_SHA} .
    - docker push ${CI_REGISTRY_IMAGE}/${ARCH}:${CI_COMMIT_SHA}

# Build amd64 image
docker_build_amd64:
  extends: .docker_build
  variables:
    ARCH: amd64

# Build arm32v7 image
docker_build_arm32v7:
  extends: .docker_build
  variables:
    ARCH: arm32v7

# Combine both images in a manifest and publish
docker_publish:
   extends: .docker_base
   stage: publish
   script:
     - |
         docker manifest create ${CI_REGISTRY_IMAGE}
           ${CI_REGISTRY_IMAGE}/amd64:${CI_COMMIT_SHA}
           ${CI_REGISTRY_IMAGE}/arm32v7:${CI_COMMIT_SHA}
     - docker manifest push ${CI_REGISTRY_IMAGE}

Warnings

Building through the emulator is slower. Unless you absolutely have to, I would recommend only building the full manifest on master and develop branches.

Using a non-Ubuntu base image in Gitpod

Simon Emms — Sat, 30 Apr 2022 00:00:00 +0000

About 3 weeks ago, the technical lead of one of Gitpod's self-hosted customers messaged me saying:

Hey Simon. I know we have to use Ubuntu for our images, but one of our teams uses a testing framework that only runs on CentOS.

Please help.

What had happened was they had conflated two facts about Gitpod:

when running a Gitpod installation, you must run Ubuntu on your Kubernetes nodes
the Gitpod workspace images use an Ubuntu image (currently buildpack-deps:focal) as the base

However, this doesn't mean that you MUST use Ubuntu. In fact, you can theoretically use any Linux distribution as your base image (except Alpine - see #3356).

Building a CentOS base image

In principal, this is just a question of copying the Ubuntu image and changing all the apt-get install commands for yum install and changing the package names where relevant.

Take a look at the Dockerfile to see how you could achieve this. There are quite a few packages in here which make it all work:

bash-completion
other Unix shells, such as Fish and ZSH
development tools
Git and Git LFS
Docker and Docker Compose

It also creates a gitpod user with the user ID 33333 and grants it anonymous sudo access.

Using this in your project

Now we have a base image, we can use this in our project. This is as simple as defining a custom Docker image in your .gitpod.yml. Then you can create a .gitpod.Dockerfile like this:

FROM ghcr.io/mrsimonemms/gitpod-centos-base-image:latest

### Node.js ###
LABEL dazzle/layer=lang-node
LABEL dazzle/test=tests/lang-node.yaml
USER gitpod
ENV NODE_VERSION=16.13.0
ENV TRIGGER_REBUILD=1
RUN curl -fsSL https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.0/install.sh | PROFILE=/dev/null bash \
  && bash -c ". .nvm/nvm.sh \
    && nvm install $NODE_VERSION \
    && nvm alias default $NODE_VERSION \
    && npm install -g typescript yarn node-gyp" \
  && echo ". ~/.nvm/nvm-lazy.sh" >> /home/gitpod/.bashrc.d/50-node
# above, we are adding the lazy nvm init to .bashrc, because one is executed on interactive shells, the other for non-interactive shells (e.g. plugin-host)
COPY --chown=gitpod:gitpod nvm-lazy.sh /home/gitpod/.nvm/nvm-lazy.sh
ENV PATH=$PATH:/home/gitpod/.nvm/versions/node/v${NODE_VERSION}/bin

USER gitpod

You will need to copy nvm-lazy.sh into your project. You could also use a cURL command to download it from GitHub, but I couldn't be bothered for a simple app.

Once you've done that, your workspace is ready for development in Gitpod

Further work

If you're having to use this for multiple projects, you will most probably want to create either a workspace-full equivalent (or whatever languages you want to support). Creating a standalone image and publishing it to your own Docker registry isn't covered here, but isn't any more difficult than creating a Docker image.

Resources

CentOS Base Image - a Gitpod CentOS base image you can use yourselves
CentOS Node Example - an example GitHub repository with a Node application

Setting Terraform Service Principal Permissions to Work With Azure Active Directory

Simon Emms — Sun, 10 Jan 2021 00:00:00 +0000

There are certain things that, no matter how many times I do them, I always have to look up. Symbolic links, for instance, are one thing I must have done at least once a week for the past 10 years, but I just can't remember whether the /path/to/fileor the /path/to/symlink comes first (it's ln -s /path/to/file /path/to/symlinkfor the record).

Configuring the permissions for a service principal to work with Azure Active Directory is a close second. Unlike the symlink though, the documentation for this is dreadful - all the commands are there, but it's just so verbose and wordy that I always miss something.

For anyone just interested in the answer (including Future Simon), the tl;dr is here

What's a Service Principal?

From the Azure documentation:

An Azure service principal is an identity created for use with applications, hosted services, and automated tools to access Azure resources. This access is restricted by the roles assigned to the service principal, giving you control over which resources can be accessed and at which level. For security reasons, it's always recommended to use service principals with automated tools rather than allowing them to log in with a user identity.

What's So Special About Active Directory?

In the topology of Azure, the AzureRMresources and the AzureADresources occupy a different place. AzureRM (short for "Azure Resource Manager") lives under a subscription. In normal circumstances, and by default, Azure will only have a single subscription per account. All the resources (such as a virtual machine) will live in here. When you want to manage one of these resources with Terraform, simply give the service principal the appropriate permissions (I usually go with Owner) on the subscription and everything will work fine.

Active Directory is different. Logically, the Active Directory resources sit outside the subscription. These are on the account itself. That means that, if you have multiple subscriptions on your Azure account, you would still only have a single Active Directory which manages things for both subscriptions.

This means that the permissions model is different in Active Directory to the subscriptions.

This is the Identity and Access Management (IAM) page for a subscription. As you can see, the "Terraform" Service Principal has the "Owner" role. The "Access control (IAM)" blade doesn't exist for the Active Directory.

Why Would You Ever Want To Mess About With The Active Directory?

There are plenty of legitimate reasons to work with the Active Directory in Terraform. One of my most regular reasons is, when setting up a Kubernetes cluster, I also set up an "admins" and a "users" group for managing which users can get access to the cluster. Those in the "admins" group have full admin access to the cluster, those in the "users" group can only get limited access via the role-based access control (RBAC) settings. The configuration for that is fairly simple:

data "azurerm_client_config" "current" {}

resource "azuread_group" "admin" {
  name = "k8s-admin"
  description = "Admin-level members for Kubernetes"
  owners = [data.azurerm_client_config.current.object_id]
  prevent_duplicate_names = true
}

resource "azuread_group" "user" {
  name = "k8s-user"
  description = "User-level members for Kubernetes"
  owners = [data.azurerm_client_config.current.object_id]
  prevent_duplicate_names = true
}

resource "azurerm_kubernetes_cluster" "k8s" {
  role_based_access_control {
    enabled = true
    azure_active_directory {
      tenant_id = data.azurerm_client_config.current.tenant_id
      managed = true
      admin_group_object_ids = [azuread_group.admin.id]
    }
  }

  # Additional configuratio - see Terraform docs for details
}

resource "azurerm_role_assignment" "admin" {
  principal_id = azuread_group.admin.id
  scope = azurerm_resource_group.k8s.id
  role_definition_name = "Azure Kubernetes Service Cluster Admin Role"
}

resource "azurerm_role_assignment" "user" {
  principal_id = azuread_group.user.id
  scope = azurerm_resource_group.k8s.id
  role_definition_name = "Azure Kubernetes Service Cluster User Role"
}

Any user added to either of these groups will get the appropriate permissions on the Kubernetes cluster.

How To Set It Up

We'll be setting up the service principal as a Group Administrator and also giving the service principal the appropriate API access.

Setting as a Group Administrator

This gives the service principal to administer groups. This is needed to add and remove groups and assign members to it.

Log into portal.azure.com and navigate to Azure Active Directory.
Select the Roles and Administrators
Select the role Groups Administrator
Select "Add assignments" and add your service principal

Granting API Access

Important. The Terraform provider still uses the old (and deprecated) API rather than the new Microsoft Graph API. Workis happening to move over to that, but at the time of writing, is still incomplete.

Log into portal.azure.com and navigate to Azure Active Directory.
Select the App Registrationsblade
Select your service principal
Select "API permissions" from the blade on the left
Select "Add a permission" and select the legacy "Azure Active Directory Graph" at the very bottom of the page.
Under "Delegated permissions", select "Directory.ReadWrite.All" and "Group.ReadWrite.All". Then click "Add permissions" to save.
Select "Add a permission", select the "Azure Active Directory Directory Graph" again
Under "Application permissions", select "Application.ReadWrite.All". Then click "Add permissions" to save

That's it. Now when you run terraform apply, it will have the permissions to create the groups with your desired configuration. Importantly, if you terraform destroy, it will also have the permissions to delete the configuration.

Delegation Permissions

Application Permissions

Using GitLab Pages to Host a Helm Registry

Simon Emms — Sun, 27 Dec 2020 00:00:00 +0000

To see this in action, check outgitlab.com/MrSimonEmms/helm-repo

Helm is a great way of sharing Kubernetes resources and making them reusable. Thedocumentation provides a way of creating a registry using a Docker image that you can host yourself. This provides lots of functionality, such as authentication and commands to interact with it. If you have your own infrastructure and need authentication, this is a great way to start. However, if you're publishing an open-source project, or you don't need authentication then managing infrastructure is an expense and overhead you don't need.

Enter GitLab Pages.

What is GitLab Pages

GitLab Pages is a way of publishing static files to the internet. It also allows you to use any URL you want and can be configured to use Let's Encrypt TLS certificates.

As a Helm Registry is simply an index.yaml file and a collection of .tar.gz files, this makes GitLab Pages a great option for hosting your registry.

Setting Up Your Repo

To set the repository up, you actually only need three files configured.

The packages directory is used in case you want to set up a website to read the index.yaml. This is outside the scope of this post, but can be copied from my Helm Registry source code

/packages/index.html

This file is required for GitLab Pages to trigger building of the website. Even though it's not required for the Helm Registry, without it, this won't be published as a website.

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <title>Helm Registry</title>
</head>
<body>
  My Helm Registry
</body>
</html>

/packages/index.yaml

This is the contents of the Helm Registry. Eventually, this will contain a list of all the packages published to your registry.

apiVersion: v1
entries: {}

/.gitlab-ci.yml

This file controls how the GitLab CI/CD builds the package. There are two tasks here:

the add_helm_chart task is run when a trigger is received. It downloads the Helm chart, adds it to the index.yaml file and commits it to the repository
the pages task is run when a commit is pushed to the master branch. It publishes thepackages directory as your website.

stages:
  - init
  - publish

image: node

variables:
  GIT_REPO_DIR: ./git-repo
  HELM_REPO_DIR: ./packages

add_helm_chart:
  rules:
    - if: '$CI_PIPELINE_TRIGGERED == "true" && $PROJECT_CHART_REPO != null && $PROJECT_OWNER != null && $TAG_NAME != null && $CHART_DIR != null && $CHART_NAME != null'
  image: registry.gitlab.com/mrsimonemms/gitlab-ci-tasks/kubectl-helm
  stage: init
  before_script:
    - git remote set-url origin https://${GITLAB_USER_LOGIN}:${GITLAB_TOKEN}@gitlab.com/${CI_PROJECT_PATH}.git
    - git config --global user.email "${GITLAB_USER_EMAIL}"
    - git config --global user.name "${GITLAB_USER_NAME}"
    - git checkout -B ${CI_COMMIT_REF_NAME}
    - git pull origin ${CI_COMMIT_REF_NAME}
    - cd ${HELM_REPO_DIR}
  script:
    - git clone https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.com/${PROJECT_OWNER}/${PROJECT_CHART_REPO}.git ${GIT_REPO_DIR}
    - cd ${GIT_REPO_DIR}
    - git checkout ${TAG_NAME}
    - cd -
    - helm package ${GIT_REPO_DIR}/${CHART_DIR}/${CHART_NAME} -d .
    - helm repo index --url ${HELM_REPO_URL} --merge index.yaml .
    - rm -Rf ${GIT_REPO_DIR}
    - git status
    - git add .
    - "git commit -m \"chore: add ${PROJECT_OWNER}/${PROJECT_CHART_REPO} ${TAG_NAME} to Helm repo\""
    - git status
    - git push origin ${CI_COMMIT_REF_NAME}

pages:
  stage: publish
  script: mv ${HELM_REPO_DIR} public
  artifacts:
    paths:
      - public
  only:
    - master
  except:
    - triggers

Configuration

For this to work, various bits of configuration must be done:

Configure Custom URL (optional)

If you want to host this on a custom URL, you can add this in the Settings -> Pages section. Follow the instructions on screen to add the DNS records.

If you don't do this, you can use the default gitlab.io URL.

Create a Personal Access Token

Create a Personal Access Token with theapi scope selected (from the documentation, you should also be able to use the write_repositoryscope although I've not tested it with that).

Add CI/CD Variables

In the Settings -> CI/CD section for your repository, create some variables:

GITLAB_TOKEN - this is the value of the Personal Access Token above. This value should be bothprotected and masked
HELM_REPO_URL - this is the URL to host the repository on. This must be the fully qualified domain, including https:// at the start. As an example, my value is https://helm.simonemms.com. This doesn't need to be protected or masked but won't hurt if they are.

Create a Pipeline Trigger

In the Settings -> CI/CD section for your repository, create a Pipeline Trigger. A good description would be "Add Helm chart to registry", although the exact working is up to you.

Keep a note of both the project ID in the example (in the formathttps://gitlab.com/api/v4/projects/xxxx/trigger/pipeline) and the token.

Adding a Chart to your Registry

As this uses git clone to get the project, this can get any public repository or any private repo that's owned by the same user/group as the Helm Registry.

Now you've set the Helm Registry repository up, you can begin to integrate this with other repositories that contain the Helm charts you wish to publish. Ultimately, this is a simple cURL call.

This worked example will use values fromgitlab.com/MrSimonEmms/openfaas-amqp1.0-connector. You will need to replace these with your own values.

export CHART_DIR=chart # Location of the Helm chart directory in the repository
export CHART_NAME=openfaas-amqp1.0-connector # Location of the chart with the Helm chart directory
export CI_PROJECT_NAMESPACE=MrSimonEmms # In GitLab CI/CD, this is pre-filled
export CI_PROJECT_NAME=openfaas-amqp1.0-connector # In GitLab CI/CD, this is pre-filled
export HELM_REPO_TRIGGER_TOKEN=xxxxxxxx # The trigger token for the Helm Registry project (generated above)
export HELM_REPO_PROJECT_ID=123456 # The project ID of the Helm Registry project (see the trigger configuration above)
export VERSION=v1.0.0 # The branch or tag to publish

curl -f -X POST \
  -F token=${HELM_REPO_TRIGGER_TOKEN} \
  -F ref=master \
  -F variables[PROJECT_OWNER]=${CI_PROJECT_NAMESPACE} \
  -F variables[PROJECT_CHART_REPO]=${CI_PROJECT_NAME} \
  -F variables[TAG_NAME]=${VERSION} \
  -F variables[CHART_DIR]=${CHART_DIR} \
  -F variables[CHART_NAME]=${CHART_NAME} \
  https://gitlab.com/api/v4/projects/${HELM_REPO_PROJECT_ID}/trigger/pipeline

This will trigger the add_helm_chart job inside the GitLab CI/CD config. After a few minutes, you will see a new commit to the master branch and then you will find the chart added to theindex.yaml file.

That's pretty much all there is to hosting your own Helm Registry in GitLab Pages. Also, this doesn't matter if a specific version is added via the trigger multiple times - Helm will manage that for you.

An introduction to Git Rebase

Simon Emms — Mon, 16 Nov 2020 00:00:00 +0000

This post assumes some familiarity with Git. This is an advanced concept and shouldn't be tried as your first foray into Git and version control.

During development on a feature branch, there will be times when we need to update our branch because the master branch has received an update. In order to keep a linear Git history, avoid using the git merge master command.

Having a linear history is more work, but it offers some advantages over a merge:

much easier to find the true source of a commit
no "merge commit" messages taking up space
reduced opportunity for conflicts
easier to revert a commit that's not been introduced as part of a pull request

Introducing Git Rebase

The git rebase master command is what we need to use instead.

# Assume you're in a branch feature/my-wonderful-feature
git checkout master
git pull
git checkout - # or git checkout feature/my-wonderful-feature
git rebase master
git push --force

If there are any issues, you can always use git rebase --abort.

What is Git Rebase?

"This is the way it should have gone"

Git rebase is a way of rewriting your history. At a simple level, it unwinds all your commits to the last common commit with your branch, applies the master commits and then puts your commits after them.

By contrast, git merge master would create a merge commit AFTER your commits. When your feature branch gets promoted to the master branch, you will end up with merge commits polluting the history.

Using Git Rebase to squash commits

In a feature branch, there will often be commits in your log like this (git log --oneline):

067c88e gah, I'm stupid. I can see why CI broke
bf2a09e erm, not sure why CI has broken so another go
7fa9388 (feature/my-wonderful-feature): feat(some-brilliant-feat): this is a brilliant feature I've worked hard on
d4193f5 (master) fix(some-fix): some fix

We've all been there. CI/CD pipelines can be a pain to get right. These commits are fine in a feature branch, but we don't want these littering the history in master - a feature branch should be the work we did, logically separated into "good" commits (it makes finding problems later much, much easier).

It's a good idea to create a backup branch. With a Git Rebase, you are changing the branch irreparably.Run git checkout -b backup/my-wonderful-feature to create a backup.

Run git rebase -i HEAD~3 to get the 3 latest commits - the 3 can be changed to anything, but you shouldn't go beyond the last common commit (in this case d4193f5)

This will present an interactive screen that looks like this:

NB. The latest commit is at the bottom.

pick 7fa9388 (feature/my-wonderful-feature): feat(some-brilliant-feat): this is a brilliant feature I've worked hard on
pick bf2a09e erm, not sure why CI has broken so another go
pick 067c88e gah, I'm stupid. I can see why CI broke

# Rebase f4b6d01..9249bd0 onto f4b6d01 (3 commands)
#
# Commands:
# p, pick <commit> = use commit
# r, reword <commit> = use commit, but edit the commit message
# e, edit <commit> = use commit, but stop for amending
# s, squash <commit> = use commit, but meld into previous commit
# f, fixup <commit> = like "squash", but discard this commit's log message
# x, exec <command> = run command (the rest of the line) using shell
# b, break = stop here (continue rebase later with 'git rebase --continue')
# d, drop <commit> = remove commit
# l, label <label> = label current HEAD with a name
# t, reset <label> = reset HEAD to a label
# m, merge [-C <commit> | -c <commit>] <label> [# <oneline>]
# . create a merge commit using the original merge commit's
# . message (or the oneline, if no original merge commit was
# . specified). Use -c <commit> to reword the commit message.
#
# These lines can be re-ordered; they are executed from top to bottom.
#
# If you remove a line here THAT COMMIT WILL BE LOST.
#
# However, if you remove everything, the rebase will be aborted.

You can now change your history. In our example, we want to combine all the commits together - this is the fixup command. Change the file so that it looks like this:

pick 7fa9388 (feature/my-wonderful-feature): feat(some-brilliant-feat): this is a brilliant feature I've worked hard on
f bf2a09e erm, not sure why CI has broken so another go
f 067c88e gah, I'm stupid. I can see why CI broke

Now save your changes and exit. If you now look at git log --oneline, you will see that only the first commit exists. You can now run git push --force to replace your remote branch with your local branch.

You can also use this method for doing other things. Generally, I find that I use the pick, rewordand fixup commands the most, although I also find edit and squash useful for editing my history before submitting a change.

Dockerising an R App

Simon Emms — Thu, 27 Aug 2020 00:00:00 +0000

R is a statistical programming language with a huge repository of tools to crunch numbers, manipulate data and do all manner of data science tasks. I've worked with a couple of teams with data scientists who use it and love it. Which is great, except for one problem.

R is a pain in the arse to Dockerise.

One irritating, but not insurmountable problem is the lack of official templates for R. Personally, I tend to stick to the official templates built by Docker which I then extend. That way you know that they're safe to use, are built to proper standards and are kept (reasonably) up-to-date. Fortunately, there is the Rocker organisation that maintains a series of images that you can use.

However, the biggest pain point by far is dependency management and the final size of the images. By design, when using RStudio developers will typically install dependencies at runtime. Here, that's fine because it's a development environment. When you're containerising your R app, this is not acceptable as containers should be immutable, pre-compiled and fast-loading. Some of these dependencies take many MINUTES to download, compile and install.

And R dependencies are big. Really big. If you thought node_modules was big, R is something else. I recently developed a fairly simple R app for the British Red Crossand the final image size was over 2GB (yes, that's GIGABYTES ). Rocker don't provide an Alpine image which doesn't help, but I don't think that's a big problem due to the size of the dependencies and even R itself. Rocker's r-base image comes out at over 800MB. This is built on debian which is 118MB - using Alpine would only reduce that by 100MB which, seeing as the R base is over 700MB, hardly seems worth the effort.

There are other issues with R dependencies. With NodeJS you have your package.json, with Python you have your requirements.txt. R doesn't really have any matching concept (although there are some workarounds) so you have to maintain your dependencies in both your Dockerfile and where you call it in your R app.

Finally, any dependencies that you need in your OS are not installed. This is fairly standard, but the default behaviour of the installer is to exit without an error which is incredibly frustrating.

So, how do you do it then?

The key to installing the dependencies is the install2.r application which is bundled with all Rocker images. This installs dependencies from the CRAN installation repository. There is also a corresponding installGithub.r binary which installs dependencies from GitHub.

In your *.R files, you will use the library() function to call your dependencies at the top of the script. Basically, every time you use it, you need to update your Dockerfile with each dependency. Yes, it's a pain to do it each time, but that's what you have to do.

library(dplyr)
library(readr)

install2.r

As mentioned above, this doesn't error by default. It'll print the errors in the logs (along with lots of other things) so you'll never know if the build has failed. Therefore, you need to use the --errorflag with this.

There is also a --skipinstalled flag which stops reinstalling any dependency that's already present in the system.

installGithub.r

Again, make sure you pass the --error flag otherwise any errors won't break the build.

Typically, you wouldn't need to use this. I only had to use this for the Red Cross because of an bug with the latest version of Tidyverse when using Ubuntu 18.04 (which is the basis of the R image). I would only suggest using this if you need to install a specific version of a dependency, because I can't work out how to do that with install2.r.

One final note, this requires remotes to be installed. So you will need to runinstall2.r --error remotes before installing anything with installGithub.r.

Full Dockerfile example

This is an example using Shiny server.

FROM rocker/shiny:3.6.3
COPY . ./src/shiny-server

# Install any OS dependencies - this is just an example and not required for these dependencies
RUN apt-get update \
    && apt-get install -y libudunits2-dev

# List of dependencies - ensure corresponds with `library()` calls in *.R files
RUN install2.r --error \
    --skipinstalled \
    readr \
    dplyr

USER shiny

Once you're here, the usual Docker command of docker build -t r-app . will build this Dockerfileinto your image.

Live Reload for OpenFaas

Simon Emms — Wed, 12 Aug 2020 00:00:00 +0000

2020-08-15: Added GoLang example

This article assumes a degree of familiarity with OpenFaaS. I won't be covering how to get started in it or the key concepts in any detail. If you want to get started with it, please see their excellent documentation.

I am a big fan of "serverless" functions. They help your application scale to a theoretically unlimited capacity. They are so useful that there is a whole myriad of implementations - AWS, Azure and GCP all have their own native versions, you can use the Serverless framework and then there's the like of KNative. And these are just the ones I can remember off the top of my head.

One of my favourite versions is OpenFaaS by the very excellent Alex Ellis. This has the advantage of being open source, a vibrant community and really easy to get started. It's also cloud native, so you're not tied into a particular cloud provider (yes, I'm looking at you AWS/Azure/GCP) - if it runs in Kubernetes, it'll run in OpenFaaS.

One of my biggest gripes with OpenFaaS is that it can be painful developing new functions. It's fine if you want to stick it in the public Docker Registry and/or you don't mind waiting for a minute or two each time you change your code. I regularly work for companies that don't want their proprietary code being published on the internet for anyone to use.

Also, I'm impatient and hate compile time.

The official workflow is:

faas-cli build # Build the image
faas-cli push # Push to Docker Hub
faas-cli deploy # Deploy to your cluster

Each of these steps could well take in excess of 30 seconds. Painful when you're hacking away.

Docker Compose To The Rescue

There is an accompanying Git repo with this. Please check that out to see the source code - gitlab.com/MrSimonEmms/openfaas-docker-compose

Docker Compose is perhaps a little unfashionable with Kubernetes-enabled teams, but it serves a great place for efficient local development.

IMPORTANT : this development workflow exists outside OpenFaaS. It uses the template that you will use when it moves to production, but when developing your function you won't have access to things like the async workflow. However, as you would be focusing on getting your function working rather than accessing it, that's usually an easy concession to make.

The Process Explained

If you are using the Classic Watchdog, there is nothing to do here and your function will automatically reflect any changes each time you invoke the function. This is because the Classic Watchdog runs the whole command each time the function is invoked.

The OF Watchdog is the modern and (in my opinion) the better watchdog. The function runs as a single HTTP endpoint on all methods. It's better because it allows improved error handling. However, because it's an application inside the function, we need to change how that application works - basically, you need to put a file watcher in that restarts the application each time you make a change to the code.

The important one here is the fprocess environment variable. This is the command that run the application. For example, in the node12 template the fprocessenvvar is node index.js.

We also need to set the volumes and set the user to root. The reason we have to set the user to root is to be able to install any global dependencies to the container at runtime. Even though this is an anti-pattern in a production container, I would argue that this is ok in a dev-only container to avoid having to maintain duplicate Dockerfiles.

Secrets

OpenFaaS supports secrets by putting the file in /var/openfaas/secrets. Docker Compose stores secrets in/run/secrets so you will need to do some form of either/or load, appropriate to the language. I suggest putting the OpenFaaS version as the default to reduce load in the production environment.

IMPORTANT: You should never store sensitive data in a repo.

Getting Started

Let's start by creating a Makefile in the root of your project. The purpose of this is to be able to easily install all the templates found in the functions.yml file. Strictly speaking, this is optional, but it makes life a lot easier.

FUNC_FILE ?= './functions.yml'

templates:
    which faas-cli || (echo "Please install 'faas-cli' package" && exit 1)
    which jq || (echo "Please install 'jq' package" && exit 1)
    which yq || (echo "Please install 'yq' package" && exit 1)

    $(eval templates := $(shell cat ${FUNC_FILE} | yq r - -j | jq -r '.functions | values[].lang'))

    for template in $(templates) ; do \
        faas-cli template store pull $$template ; \
    done
.PHONY: templates

Run make templates to download all the templates you need.

We now need a functions.yml in the root of your project. This is what we're using as the definition of all the functions we're writing. This is the exact same format as the OpenFaaS yaml file.

version: 1.0
provider:
  name: openfaas
functions: {}

Language Implementations

Node12

functions.yml

version: 1.0
provider:
  name: openfaas
functions:
  node12:
    lang: node12
    handler: ./functions/node12
    image: node12:latest

docker-compose.yml

version: '3.7'
services:
  node12:
    build:
      context: ./template/node12
    ports:
      - 3000:3000
    environment:
      fprocess: nodemon index.js
    secrets:
      - example
    volumes:
      - ./functions/node12:/home/app/function
    user: root
    command: sh -c "npm i -g nodemon && fwatchdog"

secrets:
  my-secret:
    file: ./secrets/example

To get live reload in NodeJS we use the nodemon application. If you've ever done any NodeJS work, it's incredibly likely that you'll have used nodemon as it works really REALLY well.

Python3-Flask

functions.yml

version: 1.0
provider:
  name: openfaas
functions:
  python3-flask:
    lang: python3-flask
    handler: ./functions/python3-flask
    image: python:latest

docker-compose.yml

version: '3.7'
services:
  python3-flask:
    build:
      context: ./template/python3-flask
    ports:
      - 3001:8080
    environment:
      FLASK_APP: /home/app/index.py
      FLASK_ENV: development
      fprocess: flask run
    volumes:
      - ./functions/python3-flask:/home/app/function
    user: root

Unlike the node12 template, we don't need to install any dependencies at runtime. The fprocess needs to use flask, which is already installed as that's the framework used in this template. I've kept the user as rootfor consistency, but it probably isn't actually necessary.

GoLang-HTTP

functions.yml

version: 1.0
provider:
  name: openfaas
functions:
  golang-http:
    lang: golang-http
    handler: ./functions/golang-http
    image: golang-http:latest

docker-compose.yml

version: '3.7'
services:
  golang-http:
    build:
      context: ./template/golang-http
      target: build
    ports:
      - 3002:8080
    environment:
      fprocess: air -c /go/src/handler/function/.air.toml
      mode: http
      upstream_url: http://127.0.0.1:8082
    volumes:
      - ./functions/golang-http:/go/src/handler/function
    command: sh -c "go get -u github.com/cosmtrek/air && fwatchdog"

This one is a little more involved than NodeJS/Python because the Go template builds a binary and puts it into an empty Alpine container. We have to intercept the build process to use the GoLang template by setting the build.target parameter to build (see Docker Multi-Stage buildsfor more information). As the OpenFaaS configuration is done in the final container, we need to apply this configuration to the build target. This means setting themode and upstream_url environment variables.

Finally, this uses the Go package Air to provide the live reload facility. This requires a config file, even if there's nothing inside it. As we won't need this in production, there's a .dockerignore file to not send it to the image when building. However, the Docker Compose volumes ignores this, which is exactly what we need.

Next Steps

I want to write a version for this for each officially supported OpenFaaS template in the template store. If you want to help with that, please fork the repo and add a PR.

GoLang (now done) and CSharp are the ones I'd really like to get done as they seem to be fairly popular within the community.