Forem: Integralist

Better Fastly API clients with OpenAPI Generator

Integralist — Tue, 01 Nov 2022 11:39:39 +0000

The Fastly API is huge. We have lots of customers who want to interact with it using their chosen programming language but our small set of manually maintained clients was not sufficient to handle the job of our ever-evolving API. We needed a way to scale up our API client support, and OpenAPI was the answer.

We describe our API using the OpenAPI specification, which defines a standard, language-agnostic interface. The use of OpenAPI enables us to programmatically interact with the API metadata to produce content, code examples, tools and other valuable resources.

Here is an example of one of our API endpoints. It highlights how we define a GET request and what a successful (200 OK) response looks like:

info:
  title: Backend
  description: A backend is a server identified by IP address or hostname.
  version: 1.0.0

externalDocs:
  url: https://developer.fastly.com/reference/api/services/backend

servers:
  - url: https://api.fastly.com

paths:
  /service/{service_id}/version/{version_id}/backend:
    parameters:
      - $ref: "service.yaml#/components/parameters/service_id"
      - $ref: "version.yaml#/components/parameters/version_id"
    get:
      summary: List backends
      description: List all backends for a particular service and version.
      operationId: list-backends
      responses:
        "200":
          description: OK
          content:
            application/json:
              schema:
                type: array
                items:
                  $ref: "#/components/schemas/backend_response"
              examples:
                body:
                  value:
                    - $ref: "#/components/examples/backend_response"

HINT: To reduce duplication we often put common parameters and values into external files that can be referenced separately, we also do this for large nested sections that can otherwise make it difficult to focus on the critical points of an API’s specification (see $ref).

The problem: 700 endpoints, 7 languages

In order to enable customers to access our API within a programmatic environment, we provide API clients in multiple languages. Over time we produced multiple API clients, all varying in their level of API coverage, code quality and consistency.

There were multiple factors that meant our API clients didn’t all receive the care and attention they deserved, and so would suffer from code bugs and generally be out of date with our API. For example, we needed:

Engineers who knew multiple languages and had enough experience with them to not only be able to write code in these languages but to write idiomatic code.
The resources to continually revisit clients when the API changes, as well as the sheer amount of work to support such a large number of endpoints.

The plan

In 2020 the Developer Relations team took ownership of the API clients and we began devising a plan to make them more complete, consistent, and where possible, idiomatic. This wouldn’t be achievable by maintaining all the clients as separate projects, so the primary focus was to create and maintain common tooling to enable a broad stable of API clients to keep up with our API as it evolves.

We also aimed to allow our Fastly co-workers to easily contribute to the clients when an API changes, avoiding (a) DevRel having to try and implement all API changes into the clients, and (b) other teams having to learn different standards and structures for a bunch of different API clients in order to ship a feature - neither of which would be scalable.

The solution: OpenAPI Generator

To achieve these goals we investigated various options and ultimately decided on using a community-driven fork of the official Swagger CodeGen project: OpenAPI Generator (OAG).

OAG is a template-driven engine that can generate documentation, API clients and server stubs in different languages by parsing your OpenAPI specification.

How OAG works

OAG is a Java project that uses Mustache templates to configure each supported programming language. It provides a CLI openapi-generator-cli that will download the appropriate JAR file and invoke the java executable to run OAG.

To produce a new API client, we pass openapi-generator-cli our OpenAPI specification, tell it what language we want to use, and where to output the API client code. Here is a snippet from our Makefile that uses Docker to help get the Java CLI tool configured correctly:

LOCAL_ROOT:=/local
OPENAPI_GENERATOR_CLI_TAG=v5.4.0
OAPI_GENERATOR_CLI:=docker run --rm -v $(PWD):$(LOCAL_ROOT) openapitools/openapi-generator-cli:$(OPENAPI_GENERATOR_CLI_TAG)

$(OAPI_GENERATOR_CLI) generate \
    --global-property apiTests=$(GENERATE_TESTS_API),modelTests=$(GENERATE_TESTS_MODEL),apiDocs=$(GENERATE_DOCS_API),modelDocs=$(GENERATE_DOCS_MODEL) \
    -i $(LOCAL_ROOT)/$(BUILT_SCHEMAS_DIR)/$(COLLECTION) \
    -g $(GENERATOR) \
    -t $(LOCAL_ROOT)/$(TEMPLATES_DIR)/$(TEMPLATE) \
    -c $(LOCAL_ROOT)/$(TEMPLATES_DIR)/$(TEMPLATE).yaml \
    -o $(LOCAL_ROOT)/$(TEMPLATE)$(if [$(COLLECTION_NAME) == "fastly"],,/$(COLLECTION_NAME)) \
    >$(TEMPLATE)/.tmp.$(COLLECTION_NAME).log \
    2>$(TEMPLATE)/.tmp.$(COLLECTION_NAME).err.log

Why it’s a good choice

Unlike most other tools we investigated, OAG had good support for multiple languages while also supporting the OpenAPI 3.x specification, which is the major version our own API specification was written in (most other open-source tools at the time only supported 2.x).

OAG is also a very flexible tool allowing us to control various aspects of the generated code, as well as execute post-processors and the ability to define custom generator templates if none of the built-in templates fit our requirements.

What languages we chose and why

Prior to this initiative we had five API clients, so it was important that whatever process we came up with, would support at least Ruby, Python, PHP, Perl and Go*.

* This is our hand-coded Go client - the OAG version is still in the works.

On top of that list, we also wanted to produce greenfield JavaScript and Rust clients as these are popular languages our customers are using.

Although OAG was doing the hard work of generating API clients for lots of different languages and keeping them up-to-date with our API specifications, as you’ll see, this wasn’t a panacea.

Difficulties and solutions

There were a few issues we encountered. The first was we needed a custom build process to manage a large number of OpenAPI specification files (at the time we had approximately 120 and it was growing).

The openapi-generator-cli tool could only process one specification file at a time, which meant we needed to first aggregate our specifications into a single file (with the help of redocly).

To ensure we would be able to run the code-generation process within our Continuous Integration (CI) pipeline, we needed to use a Dockerised version of openapi-generator-cli (the coordination of which was a complex process). See the next section on how we auto-regenerate clients on spec change.

Another challenge we encountered was the quality of the community-driven language templates. For the most part, they worked but they didn’t appear to be well maintained over time, nor did every feature in our OpenAPI specification have a corresponding implementation in the language templates. In some cases, the language templates didn’t support the latest programming language version, and the generated documentation wasn’t as good as we needed it to be.

Additionally, the quality of our OpenAPI specification files would cause problems in the generator, by either triggering runtime errors in the underlying Java parser or in the generated code itself due to incompatibility with the language templates.

Each of these issues would result in hard-to-debug compilation errors within the code-generation pipeline, and so we would have to figure out which of the three potential layers the issue was stemming from:

The Fastly OpenAPI specification documents.
The openapi-generator mustache templates.
The openapi-generator language parsers (written in Java).

Sometimes we would tweak the OpenAPI specification such that the generator’s language parser would take a code path that didn’t trigger a Java error, while solving issues with the language templates would mean having to manually copy the source templates and modify them to suit our needs (this was made possible thanks to OAG supporting custom template files).

Essentially we had to iterate on both ends of the integration, a bit like trying to fit a square peg into a round hole, then gradually changing the shape of both the peg and the hole until they fit.

Whenever possible we would push fixes upstream to the open-source openapi-generator project so the wider community could benefit. This was our small way to say thank you to those who came before us and who provided us with these great tools to begin with.

We also did a lot of work to restructure the generated documentation (which to be fair was already thousands of times more detailed than what we had in our original hand coded API clients) to make it much easier for users to consume and understand the API client interface.

In some cases, we were unable to resolve the incompatibility so to help unblock ourselves we implemented a custom OpenAPI extension that integrates with our CI pipeline and enables us to exclude API endpoints at a very granular level by specifying if the exclusion should be for all endpoints within a specification file or a specific sub-section. We can also exclude all clients or just those for specific languages.

Examples of this look like

x-fastly-preprocess-exclude:
  - api-client  # exclude all clients

x-fastly-preprocess-exclude:
  - fastly-rust # exclude the rust client
  - fastly-js   # exclude the javascript client

Alpha releases

The next step after successfully generating an API client was to push the code to the relevant public GitHub repository and then publish the code to its module registry (e.g. crates.io for Rust, npmjs.com for JavaScript, pypi.org for Python etc).

We selected a target group of customers and internal stakeholders to help test out the new API clients and report any bugs in the code, invalid documentation, or missing functionality. There were a few minor issues reported but otherwise, the generated API clients were working pretty well.

Auto-regenerating the clients on spec change

The last piece of the puzzle was the ability to automatically regenerate an API client when any OpenAPI specification files were modified. We achieved this by adding two new CI pipeline flows across two separate repositories that would coordinate with each other.

At a high level, this looked like:

HINT: We use GitHub Actions for our CI/CD. You can learn more about it by reading our blog post “How Fastly deploys Gatsby CMS websites to GCS using GitHub Actions”.

We wrote a CI job within the GitHub repository containing our OpenAPI specification files. That job runs whenever changes are merged into the mainline branch and starts by validating and combining the specs.

Of course, when the API definition changes, it makes sense to regenerate all the API clients; not just because we want to keep our public clients up to date, but also because as part of reviewing open PRs, it's great to be able to grab a client and use it to actually test out your new API before we integrate its definition into our official OpenAPI spec.

GitHub Actions turns out to have an incredibly useful hook called workflow_dispatch, which allows a change in one repo to trigger an action in a different repo! We define a step that uses a conditional check to prevent it from being executed if there were no specification changes, otherwise, it triggers a workflow file within the GitHub repository that contains our API client generator logic/build process.

The CI job defined in the API client generator repo is configured with the workflow_dispatch event. It receives the branch and commit SHA from the OpenAPI specification repo, and it posts status updates back to the OpenAPI repo informing it of the build progress.

The next step in the API client generator CI job checks out the appropriate branch of the OpenAPI specification repo and uses it to build all the API clients.

Finally, the CI job sends a status update to the OpenAPI specification repo letting it know where the generated API clients were uploaded to.

Conclusion

Auto-generating our API clients has been a long project. The work started with the initial move to document our API using OpenAPI specifications. This enabled us to make a much better API reference on our Developer Hub that consumes those specification files and produces content from them.

The code generation itself hasn’t been easy: we’ve had to balance the complexity of coordinating changes across multiple repositories while handling errors over multiple layers, and producing API clients that are not only idiomatic to the language they’re built in but also correct and accessible.

That said, for us, this was a task that we needed to take to ensure we were providing our customers with the best possible tools and experience, and enabling them to interact with the Fastly platform more efficiently.

We’re now able to display code examples for each supported language in our API documentation, making it easy for customers to get started using our APIs.

Any time the API definition is improved, we end up improving the quality of the API documentation. Lastly, this work has encouraged our own developers to engage with the API definition.

Take a look at the clients and let us know what you think. We love hearing how customers are using our tools and our platform.

How Fastly deploys Gatsby CMS websites to GCS using GitHub Actions

Integralist — Wed, 31 Aug 2022 14:40:30 +0000

Developing and releasing software is a complex process. The most common solution is to use an automated system for testing, validating, and publishing software.

We’re going to look at what this means, how this is typically implemented, and learn about a platform that helps us to create these software pipelines. We’ll see the features in context by digging into how Fastly has implemented them for one of its own services.

Here is a breakdown of the content:

Tools and terminology
Coordinating complex pipelines
The 10,000-foot view
Storing your workflows
Handling events
A ‘Hello World’ job
Validating against a matrix
Third-party actions
Accessing contextual data
Defining environment variables
Managing secrets
Flow control
Persisting data
Reusable workflows
Conclusion

Tools and terminology

First, let’s understand what it is we’re going to be talking about: CI/CD.

CI (Continuous Integration) is the process of automating the merging of new or updated code into an existing code base and utilising supplementary tools, such as code linters and unit tests, to help make the integration as correct as possible.

CD (Continuous Deployment) is an extension of CI, focused on automating the deployment of a piece of software once the CI process has been completed successfully.

There are many CI/CD solutions available, such as Jenkins and Travis CI (both of which Fastly has used in the past) but here I’m going to be talking about GitHub Actions, which has proven to be a powerful and flexible tool that has enabled us to simplify our content publishing pipeline.

The GitHub Actions platform is managed by GitHub (the largest online collaboration platform where developers and companies build and maintain their software). This means GitHub Actions exist alongside where our code lives, and this can enable better integration with our internal repositories compared to external solutions such as Jenkins/Travis.

Different CI/CD platforms use different terminology to describe their service model. With GitHub Actions, everything starts with a ‘workflow’. A workflow is a YAML configuration file that consists of

Jobs: a job represents a collection of 'steps'.
Steps: each ‘step’ executes logic to achieve some goal.
Events: an event determines when your jobs are run.
Runner: a virtual machine that runs your jobs.

GitHub provides a nice visualisation of this structure...

Coordinating complex pipelines

We use CI/CD across most projects at Fastly, both internal and public. I’m going to detail some examples of how we configure this for our Developer Hub (DevHub).

The DevHub is a static website generated using Gatsby, and it provides Fastly's developer community with information on, amongst other things

Fastly public APIs, and language references for VCL and Compute@Edge services.
Fastly tools that simplify the setup, testing, validation and deployment of your code.
A plethora of solutions, examples and architectural concept information.

We’re going to dig into DevHub's multi-layered CI/CD process and see how we have utilised the various features of the GitHub Actions platform to support some of its more complex requirements.

The 10,000-foot view

Let’s start with a visualisation of what we’re dealing with when it comes to DevHub’s CI/CD pipeline. Below is a “summary view” of our DevHub pipeline that the GitHub Actions UI provides. It is an interactive graph that lets us interact with each of the available jobs and to view their individual progress and status.

Here we can see we start with two consecutive jobs, i.e. one must complete before the other

Gatsby Build
Deploy to GCS

These are consecutive because we have to build our application before we can deploy it.

This pipeline is the same for both our development and production environments. The only difference is the subdomain to which the Gatsby application is deployed.

When a PR is opened on our DevHub GitHub repository, the code is deployed to a dynamically generated subdomain for QA. When the PR is merged the deployment is to our production endpoint developer.fastly.com.

After the first two consecutive jobs, we have multiple jobs that are run in parallel, handling things like checking for stale content that needs to be reviewed, updating search metadata, and purging the Fastly CDN to ensure users see the latest content.

HINT: Jobs are by default run in parallel. If a job has a dependency on another job, this can be made explicit by using jobs.<job_id>.needs.

We then introduce a blocking job called “Post-deployment checks” that validates whether any updated files are related to our Compute@Edge services, or if any of the code examples for the various Compute@Edge SDK languages have been updated. If they have, the final set of jobs will be executed, which deploys the updated service code, and validates our code examples are correct.

Some of the jobs don’t need to run every time. For example, there are jobs defined that only run on a predetermined schedule and others that only need to run when we're building the production environment (i.e. are skipped as part of the PR review process). We’ll see how this is handled later on.

Storing your workflows

Defining pipelines with GitHub Actions requires defining jobs within a workflow file. You can have any number of workflow files, which are stored in a .github/workflows folder within the root directory of your project repository.

A tree view of this directory structure might look like this:

├── .github
│   └── workflows
│       ├── my-first-workflow.yaml
│       ├── my-second-workflow.yaml
│       └── my-third-workflow.yaml

Handling events

The jobs you define within a workflow file are triggered based on specific events. GitHub Actions offer a rich set of events that let you fine-tune your workflow using the on key for numerous use cases. Some common examples of triggering a workflow are:

When pushing changes to any branch

on: push

When pushing changes to specific branches

on:
  push:
    branches:
      - "my-feature-branch"
      - "my-other-feature-branch"

When pushing a tag

on:
  push:
    tags:
      - 'v*'

Scheduling execution at a specific time or interval

on:
  schedule:
    - cron: "0 0 1 * *"   # https://crontab.guru/every-month
    - cron: "*/5 * * * *" # https://crontab.guru/every-5-minutes

HINT: https://crontab.guru/ makes dealing with the cron syntax easy.

A 'Hello World' job

Before we jump into more detailed examples, let’s look at a simplified workflow file.

name: My Workflow
on: push
jobs:
  a-simple-job:
    runs-on: ubuntu-latest
    steps:
      - name: Say Hello
        run: echo 'hello'
      - run: | 
          echo 'foo'
          echo 'bar'
          echo 'baz'

We’ve already seen the on key, but now we have the basic building blocks that you’ll see in nearly all types of workflows.

name (optional): the name of your workflow file.
jobs: a collection of jobs we want to have executed.
jobs.<job_id>: a unique job identifier (a-simple-job in our example).
jobs.<job_id>.runs-on: indicates which virtual machine to run our job on.
jobs.<job_id>.steps: a collection of steps we want to have executed.
jobs.<job_id>.steps[*].name (optional): the name of your step.
jobs.<job_id>.steps[*].run: a command-line program to execute.

Each run key represents a new process and shell. In the above example, we can see two separate steps defined, and the latter step uses a pipe character | to configure multiline mode. This means each of the echo commands are run within the same shell instance.

NOTE: the use of the multiline pipe character requires the following lines to be indented.

The caveat to using multiline mode is that any errors that occur during runtime execution become a lot harder to identify and debug compared to using individual steps. By splitting multiple commands across separate steps, any failure that occurs will be associated with a specific step.

There are many other steps[*] keys available, and we’ll see examples of them as we dig more into the DevHub’s example configuration.

When defining multiple jobs within a workflow we’re able to control the order of the jobs. By defining jobs.<job_id>.needs and providing it with the relevant jobs.<job_id> we’re able to inform the GitHub runner that we need a specific job to have been run and completed successfully before the current job can begin.

Validating against a matrix

For the DevHub we have jobs.<job_id>.runs-on set to run our workflow on a Ubuntu virtual machine but this particular key is much more powerful when coupled with jobs.<job_id>.strategy, which enables you to configure your job to be run on multiple virtual machines. An example of this configuration would be:

jobs:
  example_matrix:
    strategy:
      matrix:
        os: [ubuntu-latest, macos-latest, windows-latest]
    runs-on: ${{ matrix.os }}

The runs-on key example above is using an Expression syntax we’ve not yet discussed. It’s a feature you’ll use a lot when writing your own workflows. In summary, it lets you dynamically reference other keys and exposed data, and we’ll cover this in more detail in the section on “Accessing contextual data”.

You can even extend this approach to support a multi-dimensional matrix, enabling you to validate a job not only against multiple platforms but also using different software versions across those platforms. We use this approach in our open-source Fastly CLI pipeline when validating the CLI test suite, which needs to produce an executable binary that works across all three major operating systems (Linux, Mac, Windows).

Refer to the documentation for further examples of this type of use case.

Third-party actions

Actions are a collection of steps that can be reused. You can create your own custom actions but it’s more common to use third-party community actions, for example, Fastly provides actions for building and deploying to Compute@Edge. To use an action, the workflow job needs a step that includes the jobs.<job_id>.steps[*].uses key set to an appropriate configuration path.

A common example is using actions/checkout, which lets a job access the project code inside the repository running the workflow.

jobs:
  example:
    runs-on: ubuntu-latest
    steps:
      - run: ls # no files/directories available
      - uses: actions/checkout@v2
      - run: ls # lists all root project files/directories

The DevHub makes heavy use of third-party actions. Here are just a few of them:

actions/checkout: access repository files.
actions/cache: cache dependencies and build outputs.
actions/setup-go: install and configure a specific version of Go.
actions/setup-node: install and configure a specific version of Node.js.
actions-rs/toolchain: install and configure a specific version of Rust.

WARNING: All of the above actions, with the exception of the last one, are officially maintained by GitHub. Fastly trusts using these actions as they are developed by the same organisation providing the platform. Any non-official third-party actions should be audited first before using as they can access an automatically generated GitHub token which will enable an action to carry out API requests that otherwise require authentication. See “Managing secrets” for more information.

Accessing contextual data

GitHub Actions offers a collection of ‘context’ objects that provide information for things like your workflow environment, github repository, job and step output, secrets data and more.

We use a lot of these context objects in the DevHub workflow to generate cache keys, persist data between jobs, access secrets and other environment variables so we can configure the shell scripts that we run, and identify when a particular flow control should be applied.

An example of this is when we need to authorise access to private NPM packages hosted on GitHub’s package repository:

- run: echo "//npm.pkg.github.com/:_authToken=${{ secrets.GITHUB_TOKEN }}" > ~/.npmrc

Here we use the secrets context to access an authentication token and persist it temporarily into a ~/.npmrc file used by NPM (we’ll revisit this example later when discussing secrets in more detail).

Another example is when we use the github context to access the root directory of our checked-out project so we can access a Google authentication configuration file:

env:
  GOOGLE_APPLICATION_CREDENTIALS: ${{ github.workspace}}/.google.json

Defining environment variables

These are typically used in CI/CD to inject values at runtime. GitHub Actions lets you configure environment variables in multiple sections within the workflow file, depending on the scope they should have.

Globally: available to all jobs (and all steps within those jobs).
Job level: available to all steps within the specific job.
Step level: available for a single step.

Here is an example that demonstrates all three scope levels, they each use the env key to contain the variables being declared:

env:
  GLOBAL: foo
jobs:
  my-job:
    runs-on: ubuntu-latest
    env:
      LITERAL: bar
      INTERPOLATION: ${{ github.ref_name }}
      EXPRESSION: $(echo ${{ github.ref_name }} | perl -pe 's/[^a-zA-Z0-9]+/-/g')
    steps:
      - name: Print Global Environment Variables
        run: echo $GLOBAL
      - name: Print Job Environment Variables
        run: |
          echo ${{ env.LITERAL }}
          echo ${{ env.INTERPOLATION }}
          echo ${{ env.EXPRESSION }}
      - name: Print Step Environment Variables:
        env:
          STEPVAR: my step
        run: echo ${{ env.STEPVAR }}

We can see at the top of the example we have an env key where we define an environment variable called GLOBAL. This environment variable is declared in the top-level global scope and so it’s available at all other levels including job and step levels.

Next, we can see an env key defined within the job itself, and within that are three separate environment variables declared LITERAL, INTERPOLATION and EXPRESSION. These variables are available for all the steps within that specific job.

Lastly, we can see an env key defined within the final step, and within it is a single environment variable declared STEPVAR, which is only available to that step and no other.

You’ll notice we’ve used two different ways to access the environment variables

$NAME
${{ env.NAME }}

The first was using a syntax familiar to developers who work within a terminal environment: echo $NAME. This causes the shell instance (that the echo command runs inside) to evaluate $NAME and look up its value from the containing environment.

The other approach is to use the interpolation feature we looked at earlier: echo ${{ env.NAME }}, and although the result is the same, the way the values are acquired is subtly different. With interpolation, the value is acquired by looking up the relevant key within the env context object, and then the value is injected into the shell command to be executed (as if the command was literally echo my step), while the more traditional shell resolution approach works by the shell instance looking up the environment variable to access the value.

HINT: I would tend towards using the interpolation approach as it’s more explicit and makes setting environment variables much more ‘visible’ when scanning long workflow files.

In the previous example, you might have expected echo ${{ env.EXPRESSION }} to have printed the result of the expression (i.e. inspect the github.ref_name context and then use Perl to normalise the value), instead the literal value defined would have been printed.

This is because the env key does not evaluate shell commands like jobs.<job_id>.steps[*].run does, and causes problems when we need an environment variable to contain a dynamically generated value. So how do we do that?

Well, let’s consider the scenario we had with the DevHub. We were using the third-party action setup-node to install and configure the Node.js programming language. This action lets you specify the node version to install but it can’t be a dynamically acquired value. You either have to hardcode it or interpolate the value.

In the DevHub we have a .nvmrc file that indicates the supported node version. We want to read the version contained in this file and pass that to the action. There are a few ways to achieve this but the simplest one was to store the value in an environment variable and interpolate the value in the action’s node-version input field using the env context object.

steps:
  - run: echo "NODE_VERSION=$(cat .nvmrc)" >> $GITHUB_ENV
  - uses: actions/setup-node@v2
    with:
      node-version: "${{ env.NODE_VERSION }}"

The way this works is we use a shell command to read the node version from the .nvmrc file and assign it to a variable called NODE_VERSION and finally append that generated string to a file that the GitHub Actions runner uses to generate the environment variables. The file path is provided via the default environment variable GITHUB_ENV.

Managing secrets

A GitHub repository can be configured to store secrets. These secrets are exposed to your GitHub Actions workflow via the secrets context object. We saw an example of this earlier when we referenced secrets.GITHUB_TOKEN as a way to access our private NPM packages hosted on GitHub’s NPM package repository.

That particular secret is a special case - it's always available, since it's provided by the GitHub Actions runner. To quote GitHub directly…

At the start of each workflow run, GitHub automatically creates a unique GITHUB_TOKEN secret to use in your workflow. When you enable GitHub Actions, GitHub installs a GitHub App on your repository. The GITHUB_TOKEN secret is a GitHub App installation access token. You can use the installation access token to authenticate on behalf of the GitHub App installed on your repository. The token’s permissions are limited to the repository that contains your workflow.

That last sentence is the important bit. It means yes you can use secrets.GITHUB_TOKEN to make GitHub API calls on behalf of your repository but you can’t use it to access any other private repository in your organisation or to access GitHub’s package repository. For that we need a PAT (Personal Access Token).

This was a real problem we stumbled into and so for the DevHub it meant we had to create a PAT with the appropriate access permissions, and add it as a repository secret that we could reference via the secrets context.

Flow control

GitHub Actions provides a mechanism for preventing a job from running unless a specific condition is met. To do this you must assign an expression to jobs.<job_id>.if.

jobs:
  example-job:
    if: ${{ github.ref_name == "main" }}
    runs-on: ubuntu-latest
    steps:
      - run: echo hello

The above example looks up github.ref_name and only runs the job if the branch being run is main. The DevHub uses conditional execution for multiple use cases, such as

Only run gatsby build (an expensive operation) if we have a cache miss.
Only recompile the DevHub Compute@Edge service if it has been updated.
Only run Compute@Edge language tests if relevant code examples were updated.
Only publish API updates to Postman if it’s a scheduled production event.

HINT: The if expression can omit the outer ${{ }} but I tend to include them because I prefer the visual explicitness.

A more complex conditional used in the DevHub is when the build artifacts are deployed. We check if there were any changes to specific files, and if there were changes, we execute a job to validate those files. We do this because there’s no point in running scripts to validate files that haven’t been changed.

if: ${{ contains(needs.post-deployment-checks.outputs.data, 'RUST_CODE_SAMPLE_CHECKS=true') }}

This examples uses the contains function to inspect data exposed by another job (post-deployment-checks). We’ll learn about how the persisting of data between jobs can be achieved in the next section.

To learn more about the available operators, like ==, != and &&, refer to the operators documentation.

Persisting data

Each workflow job is executed within its own runner. This means any files or data that are created will be lost once the job finishes. There are a few ways to persist data:

Caching: actions/cache.
Artifacts: actions/upload-artifact, actions/download-artifact.
Outputs: jobs.<job_id>.outputs.

Caching

Caching is the simplest, and most common, way to persist data but it can result in confusing failure scenarios. This has been experienced with the DevHub pipeline on a few occasions. Typically a subtle workflow configuration change will cause the cache action to look up the wrong key and subsequently we either get back stale data or in some cases no data, both can cause confusing errors in the pipeline.

Caching requires two stages of configuration. You first need to define a step that indicates what to cache and what the cache key should be, then you need a separate step in another job to extract the persisted files from the cache.

# cache our content
- uses: actions/cache@v2
  with:
    path: path/to/be/cached
    key: ${{ runner.os }}-my-cache-key

# restore from cache (in a separate job)
- uses: actions/cache@v2
  with:
    path: path/to/be/cached
    key: ${{ runner.os }}-my-cache-key

HINT: You can specify multiple paths to be cached/restored using the multiline character |.

Whenever the action step is executed it will attempt to lookup the given key in the cache and if cached content is found the content is restored to the given path(s), otherwise it does nothing. When the job completes there is a ‘post run’ hook that each installed action can execute and for the cache action, when the hook is triggered, it will look at the given path and store whatever it finds there into the cache.

The DevHub uses the cache action for caching our node_modules directory (installing node modules is a very slow operation and so the less we have to do that the better), caching build configuration and resulting artifacts.

Artifacts

Artifacts are much slower at storing files than caching because the files need to be uploaded and downloaded from GitHub’s servers. Unlike caching you can only download a file that was uploaded as part of the same workflow run, but like caching they have two distinct steps that need to be defined:

# upload our files
- uses: actions/upload-artifact@v3
  with:
    name: my-artifact
    path: my_file.txt

# download our files (in a separate job)
- uses: actions/download-artifact@v3
  with:
    name: my-artifact

GitHub’s own recommendation for which approach to take (caching or artifacts) is:

Use caching when you want to reuse files that don't change often between jobs or workflow runs, such as build dependencies from a package management system.
Use artifacts when you want to save files produced by a job to view after a workflow run has ended, such as built binaries or build logs.

Outputs

Jobs have an jobs.<job_id>.outputs key which can produce data for another job to consume. This approach is used heavily in the DevHub workflow to better support caching of expensive resources such as installing node dependencies or running a Gatsby build.

The DevHub workflow dynamically generates cache keys for resources that we would like subsequent jobs to use, and it uses a job’s output key to persist those cache keys to the next job so that it can look up the cached content using the same keys.

An example of this is:

# job 1
example-job:
  runs-on: ubuntu-latest
  outputs:
    cache_key: ${{ steps.example-build.outputs.cache_key }}
  steps:
    - name: Run a build
      run: make build # generates a ./build directory
    - id: example-build
      run: echo "::set-output name=cache_key::${{ hashFiles('./build') }}"
    - name: Cache the build
      uses: actions/cache@v2
      with:
        path: ./build
        key: ${{ runner.os }}-${{ needs.example-job.outputs.cache_key }}

# job 2
other-job:
  runs-on: ubuntu-latest
  needs: example-job
  steps:
    - name: Extract the build from the cache
      uses: actions/cache@v2
      with:
        path: ./build
        key: ${{ runner.os }}-${{ needs.example-job.outputs.cache_key }}

In the first job (example-job) we see the outputs key is defined with a nested cache-key set to evaluate the expression steps.example-build.outputs.cache_key. This expression results in assigning the output from the step with id: example-build.

The outputs.cache_key in the expression is a reference to the output from the example-build step. The step’s run key executes the echo command and produces output that is structured in a specific format that GitHub Actions recognises and allows the command to communicate with the runner machine.

In this case, the command outputs a string containing the ::set-output workflow command followed by name=cache_key and assigns the named key a value which is the result of hashing the ./build directory (generated in the previous step name: Run a build) using the hashFiles function.

The digest that is generated is assigned to the example-build step output and is exposed to other steps (and the job) using the name cache_key. The job itself ensures the cache_key value from the step is assigned to its own outputs.cache_key.

The second job (other-job) states that it needs: example-job. Once that dependency is declared, other-job can access the output from example-job using the needs context object referenced using the expression syntax ${{ … }}. In this case, other-job acquires the persisted cache key with needs.example-job.outputs.cache_key and passes the digest value to the actions/cache action so it may lookup the ./build directory and restore it.

Reusable workflows

The DevHub pipeline has a bunch of ‘post-deployment’ jobs that validate the content produced by the Gatsby build system. For example we run scripts that check for broken links, and scripts that run test suites against the different Compute@Edge programming language examples provided on the DevHub site.

Each validation process is defined as a separate job. The steps for these validation jobs are typically the same but the script that is executed will be unique to the specific job, and this means all the steps that are required to enable the validation process have to be duplicated across each validation job. To help reduce the duplication of steps we define a set of reusable workflows.

A reusable workflow is a yaml file that has a structure very similar to the standard workflow yaml file we’ve seen so far. For example, you define a jobs key which determines what platform the job should run on, any environment variables that need to exist, and a set of steps that should be executed.

NOTE: Reusable workflows don’t inherit the parent workflow environment.

The only difference is that a reusable workflow file is defined as being a ‘template’ and it has access to an extra ‘event’ called a workflow_call. Within the workflow_call key you can define a collection of inputs and (optionally) secrets.

The workflow_call event is triggered by the main workflow file when it defines a job that references the reusable workflow via the uses key. The main workflow file is able to pass its own values for the defined inputs and secrets.

An example reusable workflow might look something like this:

on:
  workflow_call:
    inputs:
      script:
        required: true
        type: string
    secrets:
      api_key:
        required: true

jobs:
  template:
    runs-on: ubuntu-latest
    env:
      ...
    steps:
      - ...

The inputs and secrets can be referenced within the reusable workflow’s steps using the expression syntax ${{ inputs.script }} and ${{ secrets.api_key }}.

The skeleton outline of the DevHub’s main workflow, with a call to the reusable workflow, looks like this:

jobs:
  build:
    ...

  deploy:
    ...

  validate-foo:
    needs: deploy
    uses: <org>/<repo>/.github/workflows/<filename>@<branch>
    with: 
      script: ./foo.sh
    secrets: 
      api_key: secret_squirrel

  validate-bar:
    needs: deploy
    uses: <org>/<repo>/.github/workflows/<filename>@<branch>
    with:
      script: ./bar.sh
    secrets: 
      api_key: secret_squirrel

Reusable workflows help reduce boilerplate by abstracting away common steps. Without this feature the DevHub post-deployment validation jobs would easily have caused the overall workflow file to quadruple in size and complexity, and also would have resulted in a maintenance nightmare if ever we needed to change a set of steps, as we otherwise would have had to remember to change them in multiple places instead of just one reusable workflow file.

Conclusion

GitHub Actions is a CI platform that offers a very flexible and expressive configuration model, supporting a wide-ranging eco-system of community-driven third-party tools, all for free and directly integrated with where your code likely already exists, on the largest code repository platform available today. Try it out for yourself and see how you might simplify your existing CI workflows and take advantage of its rich feature set.