Forem: Spiroman

CircleCI - The Matrix

Spiroman — Wed, 06 Oct 2021 16:23:23 +0000

Technical examples - The matrix

This series of posts will focus on some tips and tricks that you can, and should implement in your config files. First, we will cover the matrix.

This post assumes you have the basics down about circle CI configs, i.e: you know what: job, step, parameter, etc. mean.
If you’re a complete beginner I would suggest you look up something beginner-friendly, that explains these topics (the official Circle documentation for example). I’m not saying this will be impossible to follow, but it is better if you get the full picture.

For each topic, I will post links to official documentation where you can read all the technicalities.

Matrix basics

The matrix is there to help you avoid repeating yourself, and run similar jobs that only differ from one another by a few lines.
We will cover multiple configurations, building upon our first example:

workflows:
  test-and-build:
    jobs:
      - test:
          matrix:
            parameters:
              service: ["svc1", "svc2", "svc3"]
jobs:
  test:
    parameters:
      service:
        type: string
    steps:
      - step1:
          service: << parameters.service >>
      - step2
      - step3

This will create 3 jobs that will run in parallel, and each will be passed a different value for the service parameter. I also included an example of such a job, where the first command accepts the service parameter. Now, you might ask yourself why would I need to do that?

Here’s why (this is just one example out of many): let’s say that you are working with a monorepo that contains multiple services (or projects, depends on how you call them - I will call them services) and for each of them, before deploying, you would like to install dependencies, lint, test, and maybe “Dockerize”.

Now, if you’re not doing this with a matrix, you can either copy and paste 3 jobs that only differ by the name of the service you’re currently doing all of the above for, or you’ve created a very long job that does all of that for each of them sequentially (please don’t do that).

In contrast, if you’re using a matrix, you would only need to write one job that accepts as a parameter which service to work on. Moreover, when you expand and would like to test another service, all you would need to do is expand that list (service field in the matrix).

Here’s what it would look like in a case you don’t use matrices:

workflows:
  test-and-build:
    jobs:
      - test1
      - test2
      - test3
jobs:
  test1:
    steps:
      - step1:
          service: "svc1"
      - step2
      - step3
  test2:
    steps:
      - step1:
          service: "svc1"
      - step2
      - step3
  test3:
    steps:
      - step1:
          service: "svc2"
      - step2
      - step3

This doesn’t look terrible, yet. But it’s only because this is a basic example, and because we’re using our own commands (will cover in another post). The moment you need to test a new service, you will have to copy an entire such block. Same with when you would want to add a new step to your testing job, you will have to add it to every single one… It gets time consuming, trust me.

Multi dimensional matrices

Let’s expand our previous example and look at something like this:

workflows:
  test-and-build:
    jobs:
      - test:
          matrix:
            parameters:             
              service: ["svc1", "svc2", "svc3"]
              os: ["linux", "macos"]

Now, all we’ve done is add another parameter to the matrix, but the implications are amazing. What will happen now, is that we will get 6 jobs (Cartesian Product of parameters), and can now test each service on two different OS’s. No need to write the same job over and over. Pretty nifty eh?
Please note that your matricized job cannot expand into more than 128 jobs!

Dependencies between matricized jobs

Continuing with our first example, we will look at two different ways to “wait” for these jobs to complete.

Example 1:

workflows:
  test-and-build:
    jobs:
      - test:
          name: test-<< matrix.service >>
          matrix:
            parameters:             
              service: ["svc1", "svc2", "svc3"]
      - build:
          name: build-<< matrix.service >>
          requires: test-<< matrix.service >>
          matrix:
            parameters:             
              service: ["svc1", "svc2", "svc3"]

Example 2:

workflows:
  test-and-build:
    jobs:
      - test:
          matrix:
            parameters:             
              service: ["svc1", "svc2", "svc3"]
      - build:
          requires: test
          matrix:
            parameters:             
              service: ["svc1", "svc2", "svc3"]

Let’s examine the differences between the two:

In the first example, we’re assigning a name to each of the testing jobs i.e: “test-svc1”, “test-svc2”... Following that, each of our 3 build jobs will await for its corresponding test job to end. Meaning that “build-svc1” will wait for “test-svc1”, and so on.
Please note that in order to wait for each job individually, you must name them.

In the second example, we’re waiting for all 3 of the test jobs to be complete before starting our build jobs.

Both ways are valid, and it’s up to you to decide which one you need (In some cases, a composition of the two methods could be needed).

Anchors and aliases

One last thing I want to cover is anchors and aliases. Let’s see how we can modify one of the previous examples to avoid repeating ourselves too much:

workflows:
  test-and-build:
    jobs:
      - test:
          name: test-<< matrix.service >>
          matrix:
            parameters: &params            
              service: ["svc1", "svc2", "svc3"]
      - build:
          name: build-<< matrix.service >>
          requires: test-<< matrix.service >>
          matrix:
            parameters:             
              <<: *params

As you can guess, this will function similarly to our previous example. The only difference is that we don’t have to add another service to all the jobs that use the service parameter. We’ll only need to add new services to the first map, after which all the other references will have the modified map as well.

What's next

This is the second post of the series which deals with the matrix. Read the other posts to see more technical examples.
Cheers!

Battle-tested CircleCI tips and tricks

Spiroman — Wed, 06 Oct 2021 16:11:38 +0000

Intro

Hello, everyone!
In this series of posts, I will cover a variety of topics that I think everyone who works with CircleCI (Circle) should know. For all the posts in this series, I’m assuming that the reader is familiar with the general concepts of Circle. If you’re not - I suggest you read up on such concepts as workflow, job, step, command, and other basic terms.
You could still follow along, but some things might not be clear right off the bat.

Circle CI config structure

First, we’ll talk about the structure of your config file (AKA the CircleCI YAML). It should look something like this:

Where each part of the Config file signifies the following:

Config - This part will specify the version of your CircleCI config. Any Orbs, and their versions (I’ll talk about orbs later), and any parameters you wish to define “CI wide”
Workflow - The main component of every Circle config, this is our entry point where we define workflows (there can be multiple; nightly, tests, etc). Each workflow is composed of multiple jobs.
Jobs - A series of steps/commands that will be executed on a specified machine. You can write out your own steps, or use predefined commands.
Commands - Those are steps that you have previously defined and will reference in your jobs. They are responsible for the actual work, they will have your shell commands that you wish to execute.

I have ordered them in this specific order to make it easier to read and understand what’s going on (you’re free to order them as you like).

The first thing you want to see is your configs for the run in order to see which orbs are present and what parameters are defined.
Next, the workflows. Your first level of abstraction. Just from the name you will see what their purpose is and what jobs they run.
Jobs are your second level of abstraction, they’ll have the actual commands, and you’ll be able to see what’s executed and in what order.
Lastly, the commands. There’s no abstraction here, this is the actual stuff that gets run (test, compile, build, push, etc).

Orbs

Orbs are another thing I want to talk about. They are:

“shareable packages of CircleCI configuration you can use to simplify your builds” - CircleCI.

I strongly suggest that before you start writing your own steps inside any job you look at what’s available online. Usually, someone already encountered the same problem you’re having and made things easier by writing an orb that you can use in your own config.
If you’re not sure what I mean by that, here’s an example: you want to build a Docker container and upload it to ECR (AWSs’ container registry). Instead of manually installing the AWS CLI on your machine and setting up your profile and writing all the commands yourself, you can use their orb that does all that for you! Amazing, I know.
There are many more use cases: Slack messaging, installing Node, Golang, etc. Go explore them yourselves.
Please note that they may not always fit exactly what you need! In some cases, it's easier to write out the steps manually than use some limited functionality that an orb offers.

Secrets

Secrets! You will inevitably want to use some secret information that shouldn't be written as plain text in your config file, this is where secrets come in. There are two types: organization, or project. Organization secrets are accessible from all projects/repos, while project-type secrets are only accessible from within the project/repo in which they are defined.
Their names are not actually secrets, but rather: Context, and Environment Variables, respectively. You can check out how to define contexts here, and env vars here.
Now, the question that comes to mind is when should I define a context secret, rather than an environment variable? Before I answer that, I want to clarify something about contexts, they are not exactly key-value pairs like environment variables are, but rather a collection of key-value pairs.
With this in mind, I can tell you my general rule of thumb for deciding which one to use. A collection of secrets, that serve the same purpose, or are going to be needed organization-wide, should be put in one context; e.g, a context named aws-access will hold: access keys, account Id, region, URLs, etc. While one-off project-specific secrets should be placed in an environment variable.

Here's a flowchart to get you started:

This will save you a lot of time if, for example, you change access keys. If you define them in a context, all you need to update is one or two fields, while if you define them as an environment variable, you will have to go from project to project updating those two values.

What's next

This is the first post of the series which deals with more "philosophical" topics. Read the next posts to see more technical examples.
Cheers!