Forem: Simone Gotti

Agola Direct Runs: Power to the Developers!

Simone Gotti — Wed, 21 Aug 2019 09:30:17 +0000

When we started drafting the big list of Agola wanted features, one of the primary requirements we heard from many developers was the ability to execute the agola Runs (or pipelines/workflows in other CI/CD tool concepts) from their local development directory while they were developing new features at any time (and without the need to commit).

There are many good reasons to ask for this:

They want to be sure that all the pipelines will finish successfully before opening a pull request or pushing to a branch (to avoid a loop of force push and wait for the CI/CD run being successful).
They want to test conditional runs (a run that have tasks that depends on diffent conditions: if it's a push to a branch, a push of a tag, a pull request etc...) and simulate the different conditions.
Usually developers are able to only execute a subset of tests locally but when they want to execute the same CI/CD pipeline that is executed by their CI/CD tools they face different problems:
- Their local environment is not a clean environment like the one where an agola run is executed (since every task is executed inside a container).
- Their laptops/desktops aren't usually powerful enough to execute it.
- A continuous delivery system is not just for tests but for a complete workflow (i.e. building images, deploying to test/staging/production etc...), so they'd like to test everything.
- Usually they end up pushing to a temporary branch (if the CI/CD tool is configured to permit this) as the unique way to test if the pipeline will work.

This looked like a great idea. It'll be great if you'll be able, at any time, with just one command, to test on a powerful system the full CI/CD runs, simulate possible conditions etc...

So we implemented this feature and called it direct runs and gave users the ability to execute a run from their local directory to the agola servers with just one command (agola directrun start). I ended up using direct runs many times in a day during the agola development.

Using direct runs.

Using direct runs is very simple. Once you have an user token for the agola service, at every time you can execute to agola --gateway-url "https://youragola" --toke $USERTOKEN directrun start.

This will push (more on how it works later) the local development repository to the agola server and start the runs defined in the agola config file like you pushed to a remote git repo connected to agola or created a pull request.
By the default it'll also push git untracked files (can be disabled with --untracked=false).

Without more options the command will act like pushing to a master branch, but you can control to which ref (and shortcuts for branches and tags) you want to push using the --ref, --branch or --tag options. To simulate a pull request you should push to a special ref using the --ref option (i.e. to simulate a github pull request you should use something like --ref refs/pull/1/head).

In this way you can "simulate" the runs and see the effect of them when pushing to a branch, pushing a tag or opening a pull request. Since in the run definition you can define some conditional tasks based on the source branch/tag/ref you can use these options to completely tests how a run will behave.

So, if, for example, you have defined a run that will do different things when pushing to a branch or when pushing a tag, you can easily test it.

You can also pass run variables using the --var or --var-file options.

Just a little demo of this in action:

How a direct run works

Direct runs rely on two main components:

An agola internal git server
The agola directrun start command

The internal git server is where the agola directrun start command will push the local development git repository.

The agola directrun start command will create a temporary git index based on the current index, the related tree object, commit it and store it in a temporary git ref in the local git repository, everything is transparent without impacting the working tree. Then it'll push this ref to the agola git server under a user repository to the required destination ref. Then agola will be instructed to create/start the runs defined in the agola config file.

Everything is git based so only the required git objects will be pushed making it really fast.

We developed a library called git-save to do this work, now it lives in the agola repository but we'd like to make it a standalone library since it can have many uses. It's inspired by the great git-wip project.

Building and pushing docker/OCI images with Agola and Kaniko

Simone Gotti — Tue, 23 Jul 2019 08:18:25 +0000

Today a common need is to build and push docker/OCI images in your CI/CD system. One of the primary issues users are facing when doing this is how to build images inside a container (Agola by default uses containers to execute run tasks). This usually requires using docker-in-docker and so privileged containers or bind mount the host docker socket. These choices can face some security issues or break the container isolation.

With Agola you can build and push docker/OCI images without breaking container isolation thanks to tools like kaniko.

There's an example repository here that contains a simple go program that will be built and put in a docker/OCI image.

Building the image

Let's walk through the first run definition in this example (the agola configuration format used in this example is the yaml format. But for more advanced definitions you could use the jsonnet format):

version: v0

runs:
  - name: build image
    tasks:
      - name: build
        runtime:
          containers:
            - image: golang:1.12-stretch
        steps:
          - clone:
          - run:
              name: build the program
              command: go build .
          # Copy the built binary and the Dockerfile to the workspace
          - save_to_workspace:
              contents:
                - source_dir: .
                  dest_dir: /bin/
                  paths:
                    - agola-example-kaniko
                - source_dir: .
                  dest_dir: /
                  paths:
                    - Dockerfile
                    - .dockerignore
      - name: build docker image
        runtime:
          containers:
            - image: gcr.io/kaniko-project/executor:debug
        shell: /busybox/sh
        working_dir: /workspace
        steps:
          - restore_workspace:
              dest_dir: .
          - run: /kaniko/executor --no-push
        depends:
          - build

This run definition will build the go program and then create an image without pushing it.

This is a run made of two tasks, the first one will checkout the code and build the go binary. Then it'll copy the binary and the Dockerfile in the run workspace. The build docker image task will execute kaniko and build the image.

The build docker image task uses the official gcr.io/kaniko-project/executor:debug image. The image with debug tag is used as it provides also a busybox shell (useful for debugging purposes). We have to override the default shell since in this image the busybox binaries are in the /busybox/ directory. Kaniko by default expects the "docker context" inside /workspace, so we'll use the working_dir task option to set it.

Building and Pushing the image

Usually building the image without pushing it to a registry isn't very useful, so let's see the changes needed to also push the image.

We just have to add few lines to the build docker image task.

Kaniko documents some ways to authenticate to gcr and aws registries and its images already include a credential helper for amazon ecr. For more information refer to the kaniko doc.

For dockerhub or your own registry you should create a docker config.json config file (inside /kaniko/.docker/config.json) with the required auth data.

version: v0

runs:
  - name: build and push image
    tasks:
      - name: build
        [...]
      - name: build docker image
        runtime:
          containers:
            - image: gcr.io/kaniko-project/executor:debug
        environment:
          DOCKERREGISTRY_URL: "https://index.docker.io/v1/"
          DOCKERAUTH:
            from_variable: dockerauth
        shell: /busybox/sh
        working_dir: /workspace
        steps:
          - restore_workspace:
              dest_dir: .
          - run:
              name: generate docker config
              command: |
                cat << EOF > /kaniko/.docker/config.json
                {
                  "auths": {
                    "$DOCKERREGISTRY_URL": { "auth" : "$DOCKERAUTH" }
                  }
                }
                EOF
          # build and push with kaniko. You can add more than one --destination option.
          - run: /kaniko/executor --destination your/image:tag
        depends:
          - build

The environment variable DOCKERAUTH is populated from a variable defined inside the Agola projectgroup/project, while, for this example, the DOCKERREGISTRY_URL value is defined in the run definition.

As a note, thanks to the powerful Agola secrets/variables system, you are able to map different variables based on different conditions (branch, tag, pull request) so you can reuse the same run definition and use different registries and auths for your development branches, pull request, master branch or tags

Introducing Agola: CI/CD redefined

Simone Gotti — Mon, 15 Jul 2019 16:24:32 +0000

There're many CI/CD tools outside, some of them are available only as SaaS (and many are free for Open Source projects but some of them are closed source), others are open source and you can install them wherever you want. So: why another CI/CD tool?

At Sorint.lab we used many different CI/CD tools for years, our Open Source projects usually use free SaaS tools, internally and from our customer we used Open Source tools installed on premise. So, in the years, we got a great deal of knowledge on many CI/CD tools and learned many of their pros and cons. In the end we always struggled to achieve some features that we considered very important for such kind of tools. That's why we created our own tool: Agola.

What are the requirements we tried to satisfy while designing and writing Agola?

Easy to install and manage.
Scalable and High Available: go from a single instance (single process) deployment to a distributed deployment.
Deploy anywhere: Kubernetes, IaaS, bare metal and execute the "tasks" anywhere (currently containers executors like docker or orchestrators and Kubernetes, but easily extensible to future technologies or VMs instead of containers).
Support any language, deployment system etc... (just use the right image)
Integrate with multiple git providers at the same time: you could add repos from github, gitlab, gitea (and more to come) inside the same agola installation.
Use it to manage the full development lifecycle: from build to deploy.
Tasks Workflows (that we called Runs) with ability to achieve fan-in, fan-out, matrixes etc..., everything containerized to achieve maximum reproducibility.
Git based workflow: the run definition is committed inside the git repository (so everything is tracked and reproducible). A run execution is started by a git action (push, pull-request).
Design it with the ability to achieve at most once runs: during a deployment to production we don't want multiple concurrent execution of the deploy...
Restartable and reproducible Runs (restart a run from scratch or from failed tasks using the same source commit, variables etc...)
User Direct Runs: give every user the power to test their software using the same run definition used when pushing to git/opening a pull request inside the Agola installation with just one command like if they were running tests locally (without requiring a super powerful workstation).
Testable "Runs" (what is a CI/CD environment if you cannot test your changes to the Runs definitions?): use the same run definition but use a powerful secrets and variables system to access different resources (environments, docker registries etc...).
Don't try to extend YAML to be a templating language but use a real templating language (as of now jsonnet) to easily generate the run configuration without side effects.
An advanced permissions system (work in progress).
Dependency Caching to speed up tasks

Runs

Runs are a workflow. Agola is a continuous Doer deeply integrated in a git based workflow. Agola (or better one of its main services: Run Service) executes "commands" in an containerized and organized way. These commands can be anything, written in any language and do whatever you want. It fits perfectly for a CI/CD system.

Runs are made of tasks that can depend on other tasks and execute only on some conditions, tasks are containerized executions made of multiple sequential steps.

Git based workflow

We deeply integrated it in a git based workflow because to achieve the best results and automation we have to track everything. A Git workflow let you track and control everything via git. The runs definition is inside the git repository so it's deeply attached to the code and can be reviewed as everything else.
If you love code review (we do), use pull/merge request to do it and agola will execute the run on the PR and provide the run status before you merge your changes to the "master" branch. If you use features branches Agola will execute the runs at every push and provide run status for that commit.
If you want to deploy to production when pushing to master new commits or tags, agola can do this.

Agola architecture

Agola is young but a big and ambitious project. In the next weeks we'll create more posts detailing its internals and architectural peculiarities.

Just as a small taste, Agola is composed of multiple services and it's extensible by writing new service on top of its base services APIs (no old style plugins!). To be fully distributed and high available some if its services use two primary components:

etcd to coordinate multiple instances (in many ways)
An object storage (as of today a shared posix fs like nfs, cephfs or an s3 like compatible storage)

Someone may ask now: "where's the database?". If you mean where's a relational (talking sql language) database that stores all the config, run data etc... the answer is: there isn't.

A good thing of developing something new is that you can try to do something different. Beside trying to improve the CI/CD experience we also wanted to experiment (and try to innovate) with some new architectural choices.

Since we want a fully distributed and high available system we wanted to rely only on components that could be fully distributed and high available: etcd and an object storage meet these requirements. So, instead of adding a third component that will increase the project requirements and that is, as of today, difficult to scale and made high available (ok there are some products that can achieve this like cockroachdb) and be free to structure our data as we preferred, we tried to achieve a atomic, consistent, isolated and transactional store using etcd and an object storage.

In this way everything is managed at the application level without putting part of the logic inside the relational database (constraints, foreign keys etc...) or relying on features that are available only on some products. But this is a big argument (and an experiment that could change or prove it wrong) that we'd like to elaborate in a future post (an hint: we are not ditching relational databases, if you look at the agola code you'll find a package called readdb that uses sqlite: yes we are using a per instance/local, rebuildable, subject to a lot of schema changes relational database as a read only database for queries).

Try it

Just try the agola demo and see how we are using it to build/test agola

The Agola documentation with various examples is in the Agola web site

Get in touch

Agola is an open source project created by Sorint.lab but we need a community around it. To get in touch for anything (contributing, help, proposals etc...) join the Agola forums and our github organization. And don't forget to leave a star on the agola main repository (https://github.com/agola-io/agola), it costs you nothing!