Forem: Lucas Jacques

Managing Firecracker microVMs in Go

Lucas Jacques — Thu, 28 Sep 2023 13:47:12 +0000

This article is the second iteration of the "Orchestrating MicroVMs, from first principles" series about the creation of an orchestrator of Firecracker microVMs in Go named Ravel. You can find the first article here.

Generally, an orchestrator cluster is made up of 2 kinds of nodes. First, there are worker nodes which handle computing workload. Secondly, there are supervisor nodes which orchestrate and schedule the workload on worker nodes.

Today we'll only talk about the worker node. Our worker will expose a REST API which will respond to the following needs:

Create microVMs from a container image,
Manage microVMs lifecycle (start, stop, delete microVMs),
List and inspect microVMs,
Expose microVMs logs (will be covered in a future article).

For the purpose of the article all the presented code is simplified and truncated of error management. You can find the full implementation on the Ravel Github repository.

From containers to microVMs

A container image (OCI image) holds all binaries required to run a workload and metadata about the intended process at container start, but it lacks a real Linux distro and kernel. Instead, it relies on the host system’s kernel, with the container runtime, like Docker or containerd, managing the container's lifecycle and interactions with the host system.

The metadata within the image, specifying entry points, environment variables, and command arguments, is crucial for initializing the container correctly by the runtime.

So, to run a container image in a microVM we need to bring a Linux kernel and an init binary to execute at kernel boot.

For now, our init binary will be mostly a copy of the thi-startup one. This needs to be mounted as a root file system and accompanied by a config file providing the metadata of the container image. Then, we extract the image content into a EXT4 file system and mount it as a second file system.

The init binary is charged to make the OCI image usable like a kind of linux distro (some linux mounts among other things) and to spawn a process with of the intended worlkoad from the provided config file.

That's about all we need to construct our microVM.

An overview of the Go implementation

Here is a simplified version of our drive creation implementation:

func (dm *DrivesManager) CreateDrive(dc RavelDriveSpec) (*Drive) {
    driveId := utils.NewId()    
    // Create a file and allocate it the desired size
    utils.Fallocate(getDrivePath(driveId), dc.Size)
    // Format it as an ext4 file system
    exec.Command("mkfs.ext4", getDrivePath(driveId)).Run()
    // Store it in a local store
    dm.store.StoreRavelDrive(driveId, dc)

    return &Drive{
        RavelDrive: drive,
    }
}

And our drive object interface:

type Drive interface {
    Mount() error
    Unmount() error
    GetMountPath() string
    GetDrivePath() string
}

Then here is how we build the VM extracted from the original file:

func (machineManager *MachineManager) buildMachine(machineSpec types.RavelMachineSpec) (types.RavelMachine, error) {
    machineId := utils.NewId()
    // Get the desired image
    machineManager.images.GetImage(machineSpec.Image)
    // Generate the config from image
    imageInitConfig := image.GetInitImageConfig()
    // Build init drive and add the init binary and config into it
    initDrive := machineManager.buildInitDrive(machineSpec, imageInitConfig)
    // Build main drive and extract container image into it
    mainDrive := machineManager.buildMainDrive(machineSpec)

    return types.RavelMachine{
        Id:               machineId,
        RavelMachineSpec: &machineSpec,
        InitDriveId:      initDrive.Id,
        RootDriveId:      mainDrive.Id,
        Status:           types.RavelMachineStatusCreated,
    }
}

Managing microVMs lifecycle

Now that we have a file system and an init, we'll use Firecracker to spawn the microVMs. Each Firecracker microVM is attached to a unique Firecracker VMM (Virtual Machine Manager) process. So when we call firecracker.NewMachine from the Go SDK, the returned *firecracker.Machine object has methods to interact with this process.

We need to track this object during all the lifetime of the Firecracker VMM which is attached to our Golang process so we'll keep it in memory with a VMMManager which is just a little abstraction over the SDK:

type VMMManager struct {
    machines map[string]*firecracker.Machine
}

type VMMManagerInterface interface {
    CreateMachine(ctx context.Context, machineId string, config firecracker.Config) (*firecracker.Machine, error)
    StartMachine(ctx context.Context, machineId string) error
    StopMachine(ctx context.Context, machineId string) error
    GetMachine(machineId string) (*firecracker.Machine, error)
    DeleteMachine(ctx context.Context, machineId string) error
}

We'll probably need to extract this code into a pluggable "driver" and communicate with it under an unix socket, or something like that, to decorrelate the microVMs lifecycle from our worker API. This will allow us to keep the machines on even when turning off the API. This interface will also probably become more hypervisor agnostic in the future.

Let's create our first microVM:

func (machineManager *MachineManager) CreateMachine(ctx context.Context, ravelMachineSpec types.RavelMachineSpec) (string) {
    ravelMachine := machineManager.buildMachine(ravelMachineSpec)

    firecrackerConfig := GetFirecrackerConfig(&ravelMachine)

    machineManager.machines.CreateMachine(ctx, ravelMachine.Id, firecrackerConfig)

    machineManager.store.StoreRavelMachine(&ravelMachine)


    return ravelMachine.Id
}

We can now start it:

func (machineManager *MachineManager) StartMachine(ctx context.Context, machineId string) error {
    machineManager.machines.StartMachine(ctx, machineId)

    machineManager.store.UpdateRavelMachine(machineId, func(rm *types.RavelMachine) {
        rm.Status = types.RavelMachineStatusRunning
    })


    return nil
}

We then do the same to stop and delete microVMs.

Here we are, so now let's expose theses features over this REST API:

GET /machines              List machines
GET /machines/<id>         Inspect machines
POST /machines             Create a machine
POST /machines/<id>/start  Start a machine
POST /machines/<id>/stop   Stop a machine
DELETE /machines/<id>      Delete a machine

To do that we use a minimal HTTP router for Go named Flow:

mux.HandleFunc("/api/v1/machines", h.CreateMachineHandler, "POST")
mux.HandleFunc("/api/v1/machines", h.ListMachinesHandler, "GET")
mux.HandleFunc("/api/v1/machines/:id", h.GetMachineHandler, "GET")
mux.HandleFunc("/api/v1/machines/:id/start", h.StartMachineHandler, "POST")
mux.HandleFunc("/api/v1/machines/:id/stop", h.StopMachineHandler, "POST")
mux.HandleFunc("/api/v1/machines/:id", h.DeleteMachineHandler, "DELETE")

And that's it, we have a REST API to manage firecracker microVMs. But it's still a work in progress...

What's next ?

There is a lot of work to make this worker ready to handle workload:

Add logs management and an endpoint for it
Add a VSOCK device to enable docker exec like stuff with the init.
Use the Firecracker Jailer to improve security
Add networking management to bring internet connectivity and, in the future, cluster wide connectivity.
Add clean OCI images management decoupled from docker.
Track status of the workload and failure recovery
Add resource management

And many other things...

I will publish more blog posts in the future for some of these points. To stay tuned, you can follow me on X (formerly Twitter).

For Potential Contributors: The world of open-source thrives on collaboration. If you're passionate about distributed systems and want to contribute Ravel welcomes you.

For Criticisms and Feedback: Every system can be refined and improved. Your constructive feedback is invaluable to the development of Ravel.

Thanks for reading.

Orchestrating MicroVMs, from first principles: Introduction

Lucas Jacques — Thu, 21 Sep 2023 14:19:18 +0000

At Valyent, our mission is to spare you the pain of deploying your applications. To do this, we build an OCI image from your code and we run it on our infrastructure.

We convey the right image to the right machine with the right compute resources for each of our customers. This is called orchestration.

Orchestration is the automation of the deployment, scaling, and operation of applications. It manages resources allocation, health monitoring, ensures proper networking, handles load balancing, and oversees application updates and rollbacks.

At Valyent we build a PaaS which could be partially summarized as a managed orchestration with good UX and good DX.

Here are the specifics needs of a PaaS:
1) Orchestrating compute resources: A PaaS must accommodate a large number of users and their computing needs, so an orchestration system is needed to support the workload efficiently.
2) Networking: A PaaS need a robust networking system to accomade internal networking between apps of an organization and external networking to allow traffic from internet.
3) Isolation & multi-tenancy: Each tenant have security and privacy needs. It is unthinkable that one tenant could access the resources of another. The PaaS must ensure the isolation between each tenant.

State of the art of orchestration

The market itself presents a pantheon of tools, each with its merits and challenges :

Kubernetes, for instance is the catch-all tool offering a lot of features but not without dragging users through intricate labyrinths. Achieving multi-tenancy in Kubernetes is hard, with its namespaces providing limited isolation, potentially leading to security and resource concerns.
Then there's Nomad, open-source, albeit with asterisks (BSL). Its single-responsibility design necessitates external tools for functionalities like service discovery and load balancing (Consul), secrets management (Vault) etc. But most of multi-tenancy features are only included in Enterprise plan which is expensive and this for each product.
While they are ready for multi tenancy, platforms like OpenStack and OpenNebula were primarily designed to manage giants clusters and weren’t precisely created for apps orchestration. Their focus on diverse workloads can make them seem like a maze with a lot of complex components.

Multi-Tenant Distributed System Bottleneck

As you have certainly understood, building a distributed system ready for multitenancy is not easy. The isolation requirements of a multi-tenant orchestrator can be divided into two main parts:

Network: Ensuring isolated and secure internal networking between the apps of a tenant is crucial.
Execution: We need to execute multiple customer, potentially unsafe, code on the same host. The application of one tenant must not interfere in any way with that of another and must only use the computing resources to which it is allocated.

In the scope of this article we'll focus on the execution part. The technology most currently used to run applications in isolation are containers, popularized by tools like Docker and widely used by Kubernetes, Nomad, etc.

The problem with containers

Containers are offering numerous benefits in terms of efficiency, portability, and modularity. However, when it comes to security, containers have intrinsic challenges that set them apart from traditional virtual machines (VMs).

Let's explore why containers might not be as secure as one might think, and how they differ from VMs in this regard:

Shared Kernel: Containers, unlike VMs, share the host's operating system kernel. This means that if a malicious actor manages to exploit a vulnerability in the kernel, all containers running on that host could potentially be compromised. In contrast, VMs have their own full-fledged OS and kernel, providing a stronger isolation boundary and thiner attack surface.
Weak Isolation: By design, containers are less isolated from each other than VMs. If an attacker can break out of a container, they might gain access to other containers or even the host system. Such container escape vulnerabilities can be catastrophic in a shared environment.
Resources management: Containers sharing a host can impact each other's performance, leading to the 'noisy neighbor' phenomenon. Memory allocation can be tricky; if a container surpasses its limit, it may be terminated. Shared kernel resources, like open file descriptors, can become exhausted by one container, affecting others. Resource isolation isn't foolproof; shared resources, such as CPU cache, can be exploited.

While containers offer a plethora of benefits like speed, they intrinsically possess vulnerabilities and challenges not present in the VM realm.

Firecracker

Enter Amazon Firecracker, a cloud-native alternative. Designed for lightweight, multi-tenant environments, Firecracker offers speed, agility, and efficiency, setting it apart from counterparts like QEMU.

So it's magic, just replace the containers with Firecracker microVMs? Not really.

First, Firecracker was designed to execute serverless tasks and is the underlying engine for Amazon Lambdas. By default, it's not really made for long-running task. Second, orchestrating firecracker microVMs is not easy. There is not as much tooling as for containers. Moreover, by default, Firecracker does not natively allow you to execute OCI images.

So what do we do next ?

Introducing Ravel, an Open-Source Firecracker Orchestrator

This post is the first iteration of a long series detailing our journey to build our own microVMs orchestrator: Ravel. Here are the main characteristics expected from Ravel:

Distributed Firecracker microVMs orchestration
OCI images compatible out of the box
Multi-tenant by default: secured private internal networking included
Easy to use and to install
Open-source

The next article will cover the implementation of the first building block of Ravel:
A small golang programm to manage multiple firecracker microVMs easily.

If you're interested by Valyent and Ravel you can join the waitlist on www.valyent.dev

Thanks for reading & let’s embark on this journey together.