Forem: Oleksandr Borodavka

Building CI/CD for Vertex AI pipelines: The production

Oleksandr Borodavka — Wed, 26 Apr 2023 12:10:08 +0000

Hi there! As you probably already know from the first few articles of this series, we tested some new ideas and tools with a POC for one simple pipeline. Today we will review a new generalized version of CI/CD for Vertex AI pipelines we have built based on that experience and some further investigations.

A bit of context

Let's recall some base points to refresh the context:

Vertex AI is used to build training pipelines.
GitHub Actions is our CI/CD tool.
We built the declarative framework that allows us to standardize the format and operations for all our components and pipelines.
The specifications and implementations of all our components and pipelines are kept in one GitHub repository.
There are three environments: development(DEV), staging(STAGE) and production(PROD).

Stating the task

What do we want to achieve?
In simple words, we want to automate everything as much as possible. Ideally, when new changes appear in our code repository we want these changes applied in production in as short time a period as possible without any manual effort. Moreover, it should be done in a stable, safe, reproducible, and effective way.

Running a little ahead, there are three absolutely awesome things in our CI/CD practice. In all deployment environments, our Continuous Integration system:

automatically rebuilds components and runs their unit and integration tests
builds pipelines changed in Pull Requests
and rebuilds dependent pipelines.

The way of code

Okay how can we achieve this?
Since developers work with the codebase in the form of Pull Requests we can use them as starting points for the workflows. There are two moments that we have to automate.

The first one is when a pull request is opened (or updated). Here we want to run the more basic and faster checks for the coming changes to provide quick feedback for a developer. These generally are unit tests and building jobs (just a build to check if configurations are okay) on our DEV environment. Then, if everything is fine, we have to run integration tests for components and run our pipelines on the STAGE environment to be sure it is ready to be merged.

The second moment is when the PR is approved and merged. Here we have the changes which have already been tested, reviewed, and merged into the main branch, so it is ready for delivery. All the processes are run again but now on the PROD environment this time.

The implementation

For both stages, we use GitHub Actions workflows and some CLI commands of the Python framework, since routines related to code analysis are more expedient to implement with the framework’s specific code. It probably doesn’t make sense to review all the code, that would be too long and too specific. However, we can take a look at the general structure and a bit of a simplified version of the workflows that have detail enough to present the idea.

Here is the structure of the source code:

pipelines/
  pipeline1.yaml
  ...
components/
  component1/
    src/
      ...
    tests/
      unit/
          ...
      integration/
          test.yaml
    config.yaml

There is a folder with pipeline specifications and a folder with components. Each component contains source files, tests, and a configuration.

And it is how the GitHub Actions directory looks like:

.github
  actions
    build_component
    build_pipeline
    run_pipeline
    test_component
    test_component_integration
  workflows
    pr_merged.yml
    pr_opened.yml

Where the actions is a directory with reusable composite actions, which do all the operations we need with components and pipelines (build, test, and run). And in the workflows directory, we have two workflows that are automatically triggered by GitHub when a Pull Request is opened/updated or merged respectively.

PR is opened

Let's take a look at the first workflow:

name: MLOps Pull Request - Opened

on:
  # The workflow will be run automatically when a PR is 
  # opened to the main branch or changes to the PR are pushed
  pull_request:
    types: [opened, synchronize, reopened]
    branches:
      - "main"
    paths:
      - "pipelines/**/*.yaml"
      - "components/**/*.yaml"
      - "components/**/*.py"

# Use concurrency to ensure that only a single workflow 
# using the same concurrency group will run at a time
concurrency: dev_stage_environment

jobs:
  git_diff:
    # Get a list of changed files in the PR for the future analysis
    runs-on: ubuntu-latest
    outputs:
      diff: ${{ steps.getter.outputs.diff }}
    steps:
      - name: Checkout
        uses: actions/checkout@v3
        with:
          fetch-depth: 0
      - name: Get git diff
        id: getter
        run: |
          GIT_DIFF="$(echo $(git diff --name-only origin/main...origin/${GITHUB_HEAD_REF}))"
          echo "::set-output name=diff::$GIT_DIFF"
  get_component_list:
    # Get a list of names for the added/changed components
    needs: git_diff
    runs-on: ubuntu-latest
    outputs:
      names: ${{ steps.getter.outputs.names }}
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Get list
        id: getter
        run: |
          NAMES="$(make get_components --paths='${{ needs.git_diff.outputs.diff }}')"
          echo "::set-output name=names::$NAMES"
  test_and_build_components_on_dev:
    # Test and build changed components on the DEV environment
    needs: get_component_list
    runs-on: ubuntu-latest
    environment: development
    strategy:
      # Use matrix strategy to run the tasks in parallel
      matrix:
        name: ${{ fromJson(needs.get_component_list.outputs.names) }}
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Test component
        uses: ./.github/actions/test_component
        with:
          component_name: ${{ matrix.name }}
      - name: Build component
        uses: ./.github/actions/build_component
        with:
          component_name: ${{ matrix.name }}

  get_pipeline_list:
    # Get a list of names for the added/changed pipelines
    needs: [ git_diff, test_and_build_components_on_dev ]
    runs-on: ubuntu-latest
    outputs:
      names: ${{ steps.getter.outputs.names }}
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Get list
        id: getter
        run: |
          NAMES="$(make get_pipelines --paths='${{ needs.git_diff.outputs.diff }}')"
          echo "::set-output name=names::$NAMES"
  build_pipelines_on_dev:
    # Build changed pipelines on the DEV environment
    needs: get_pipeline_list
    runs-on: ubuntu-latest
    environment: development
    strategy:
      # Use matrix strategy to run the tasks in parallel
      matrix:
        name: ${{ fromJson(needs.get_pipeline_list.outputs.names) }}
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Build pipeline
        uses: ./.github/actions/build_pipeline
        with:
          pipeline_name: ${{ matrix.name }}

  get_indirect_pipeline_list:
    # Get a list of names for the indirectly changed pipelines
    # (when a related component was changed)
    needs: [ git_diff, build_pipelines_on_dev ]
    runs-on: ubuntu-latest
    outputs:
      names: ${{ steps.getter.outputs.names }}
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Get list
        id: getter
        run: |
          NAMES="$(make get_indirect_pipelines --paths='${{ needs.git_diff.outputs.diff }}')"
          echo "::set-output name=names::$NAMES"
  build_indirect_pipelines_on_dev:
    # Build indirectly changed pipelines on the DEV environment
    needs: get_indirect_pipeline_list
    runs-on: ubuntu-latest
    environment: development
    strategy:
      # Use matrix strategy to run the tasks in parallel
      matrix:
        name: ${{ fromJson(needs.get_indirect_pipeline_list.outputs.names) }}
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Build pipeline
        uses: ./.github/actions/build_pipeline
        with:
          pipeline_name: ${{ matrix.name }}

  build_components_and_test_integration_on_stage:
    # Build changed components and run integration tests 
    # for them on the STAGE environment
    needs: [ build_pipelines_on_dev, build_indirect_pipelines_on_dev, get_component_list ]
    runs-on: ubuntu-latest
    environment: staging
    strategy:
      # Use matrix strategy to run the tasks in parallel
      matrix:
        name: ${{ fromJson(needs.get_component_list.outputs.names) }}
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Build component
        uses: ./.github/actions/build_component
        with:
          component_name: ${{ matrix.name }}
      - name: Test component integration
        uses: ./.github/actions/test_component_integration
        with:
          component_name: ${{ matrix.name }}

  build_and_run_pipelines_on_stage:
    # Build changed pipelines and run them on the STAGE environment
    needs: [ build_components_and_test_integration_on_stage, get_pipeline_list ]
    runs-on: ubuntu-latest
    environment: staging
    strategy:
      # Use matrix strategy to run the tasks in parallel
      matrix:
        name: ${{ fromJson(needs.get_pipeline_list.outputs.names) }}
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Build pipeline
        uses: ./.github/actions/build_pipeline
        with:
          pipeline_name: ${{ matrix.name }}
      - name: Run pipeline
        uses: ./.github/actions/run_pipeline
        with:
          pipeline_name: ${{ matrix.name }}

  build_and_run_indirect_pipelines_on_stage:
    # Build indirectly changed pipelines and run them on the STAGE environment
    needs: [ build_and_run_pipelines_on_stage, get_indirect_pipeline_list ]
    runs-on: ubuntu-latest
    environment: staging
    strategy:
      # Use matrix strategy to run the tasks in parallel
      matrix:
        name: ${{ fromJson(needs.get_indirect_pipeline_list.outputs.names) }}
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Build pipeline
        uses: ./.github/actions/build_pipeline
        with:
          pipeline_name: ${{ matrix.name }}
      - name: Run pipeline
        uses: ./.github/actions/run_pipeline
        with:
          pipeline_name: ${{ matrix.name }}

It is automatically called when a pull request is opened or updated. Due to the concurrency feature, it will be run once at a time. However, all the similar jobs, like tests will be run in parallel.

The logic inside is as follows:

Analyze the changes in a pull request and find out which components and/or pipelines were affected.
Build them in the predefined order.
Run automated tests for the components.
Run the pipelines to retrain models and deliver them for the final usage.

It is universal and works with any components and pipelines when they follow the framework agreements. Also, it is safe, works without duplicates, runs the jobs in the right order, and parallelizes them when it is possible.

The current workflow operates on development and staging environments.

PR is merged

The second workflow is run when PR is merged.

name: MLOps Pull Request - Merged

on:
  # The workflow will be run automatically when a PR is closed
  pull_request:
    types:
      - closed
    branches:
      - "main"
    paths:
      - "pipelines/**/*.yaml"
      - "components/**/*.yaml"
      - "components/**/*.py"

# Use concurrency to ensure that only a single workflow 
# using the same concurrency group will run at a time
concurrency: prod_environment

jobs:
  if_merged:
    # There is no way to trigger the workflow when it was merged 
    # (for now we know only it was closed)
    # so we have to check it at the first job
    if: github.event.pull_request.merged == true
    runs-on: ubuntu-latest
    steps:
      - run: echo The PR was merged
  git_diff:
    # Get a list of changed files in the PR for the future analysis
    needs: if_merged
    runs-on: ubuntu-latest
    outputs:
      diff: ${{ steps.getter.outputs.diff }}
    steps:
      - name: Checkout
        uses: actions/checkout@v3
        with:
          fetch-depth: 0
      - name: Get git diff
        id: getter
        run: |
          GIT_DIFF="$(echo $(git diff --name-only ${GITHUB_SHA}^ ${GITHUB_SHA}))"
          echo "::set-output name=diff::$GIT_DIFF"
  get_component_list:
    # Get a list of names for the added/changed components
    needs: git_diff
    runs-on: ubuntu-latest
    outputs:
      names: ${{ steps.getter.outputs.names }}
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Get list
        id: getter
        run: |
          NAMES="$(make get_components --paths='${{ needs.git_diff.outputs.diff }}')"
          echo "::set-output name=names::$NAMES"
  test_and_build_components_on_prod:
    # Test and build changed components on the PROD environment
    needs: get_component_list
    runs-on: ubuntu-latest
    environment: production
    strategy:
      # Use matrix strategy to run the tasks in parallel
      matrix:
        name: ${{ fromJson(needs.get_component_list.outputs.names) }}
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Test component
        uses: ./.github/actions/test_component
        with:
          component_name: ${{ matrix.name }}
      - name: Build component
        uses: ./.github/actions/build_component
        with:
          component_name: ${{ matrix.name }}
      - name: Test component integration
        uses: ./.github/actions/test_component_integration
        with:
          component_name: ${{ matrix.name }}

  get_pipeline_list:
    # Get a list of names for the added/changed pipelines
    needs: [ git_diff, test_and_build_components_on_prod ]
    runs-on: ubuntu-latest
    outputs:
      names: ${{ steps.getter.outputs.names }}
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Get list
        id: getter
        run: |
          NAMES="$(make get_pipelines --paths='${{ needs.git_diff.outputs.diff }}')"
          echo "::set-output name=names::$NAMES"
  build_and_run_pipelines_on_prod:
    # Build and run changed pipelines on the PROD environment
    needs: [ test_and_build_components_on_prod, get_pipeline_list ]
    runs-on: ubuntu-latest
    environment: production
    strategy:
      # Use matrix strategy to run the tasks in parallel
      matrix:
        name: ${{ fromJson(needs.get_pipeline_list.outputs.names) }}
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Build pipeline
        uses: ./.github/actions/build_pipeline
        with:
          pipeline_name: ${{ matrix.name }}
      - name: Run pipeline
        uses: ./.github/actions/run_pipeline
        with:
          pipeline_name: ${{ matrix.name }}

  get_indirect_pipeline_list:
    # Get a list of names for the indirectly changed pipelines
    # (when a related component was changed)
    needs: [ git_diff, build_and_run_pipelines_on_prod ]
    runs-on: ubuntu-latest
    outputs:
      names: ${{ steps.getter.outputs.names }}
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Get list
        id: getter
        run: |
          NAMES="$(make get_indirect_pipelines --paths='${{ needs.git_diff.outputs.diff }}')"
          echo "::set-output name=names::$NAMES"
  build_and_run_indirect_pipelines_on_prod:
    # Build indirectly changed pipelines and run them on the PROD environment
    needs: [ build_and_run_pipelines_on_prod, get_indirect_pipeline_list ]
    runs-on: ubuntu-latest
    environment: production
    strategy:
      # Use matrix strategy to run the tasks in parallel
      matrix:
        name: ${{ fromJson(needs.get_indirect_pipeline_list.outputs.names) }}
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Build pipeline
        uses: ./.github/actions/build_pipeline
        with:
          pipeline_name: ${{ matrix.name }}
      - name: Run pipeline
        uses: ./.github/actions/run_pipeline
        with:
          pipeline_name: ${{ matrix.name }}

This workflow is quite similar to the previous one, just runs all the jobs in the production environment.

Conclusion

That’s it! The presented solution has been working well for us for more than 6 months already. We have some ideas on how to make it even better and maybe we will share the results with the community in the next articles.

I hope this will be be useful to you in your MLOps journey and helps you to save some time while building your own CI/CD process for ML pipelines.

Please, feel free to share any thoughts, questions, or proposals in the comments.

Thank you and happy coding!

Building CI/CD for Vertex AI pipelines: The real world

Oleksandr Borodavka — Tue, 20 Sep 2022 15:11:50 +0000

Our first solution for a CI/CD implementation is a great start for us. It works well when you do not change many things at the same time when the team is small and updates are not frequent. However, the real workflow usually is not so simple. Let's take a look at some possible situations that can happen if we stay with the first implementation.

Concurrency mess

Issue

Let's look at what we get if we open a pull request with several changes. For instance, we have two changed components and one pipeline, and the pipeline contains both of these components.

In this case, GitHub Actions runs 3 workflows in parallel. Since the components are related to the pipeline, the pipeline will be built after each component. So the same job will be run 3 times. Also, we do not have any guarantee about the order of execution. It can be done in any order. Like on the diagram, firstly the pipeline will be built with component2, then with component1, and as a result, we will receive the pipeline only with updated component1, but none of the builds will be with both updated components.

Solution

We can avoid such situations by running the processes in a predefined order. In our case, there are no relations like component-to-component or pipeline-to-pipeline, we have only relations component-to-pipeline. So, if we would be able to run firstly all processes for components and after all processes for pipelines, it would guarantee us the correct build in the end.
Our solution is as follows, within one GHA workflow we do the following steps:

Analyze the list of changed files in a pull request
- Find changed components
- Find changed pipelines
- Find pipelines related to changed components
Run jobs for all component
Run jobs for all pipelines

It can be done with the matrix strategy and some additional logic in the code. The matrix strategy is a feature of GitHub Action that lets to run many jobs in parallel for a matrix of parameters. In our case, we can extract a list of changed components/pipelines and run all necessary tasks for them with a single job definition.

Doing all jobs in one workflow deprives us of duplicates as well. We also gain one more important outcome, we do not need to manually add and maintain workflows for every pipeline and component anymore as we did in the first implementation. Everything is being built on the fly.

Many Pull Requests

Issue

Great, but what if we have two or more open pull requests? Each PR is run in the required order, it is already solved in the previous step. However, since we store all the configurations and docker images in the cloud, they are shared resources. It means that one CI/CD process can interfere with another process, rewrite common resources, and so on.

Here we have a mess again. In addition, if component1 from PR1 is buggy and it is built in time between component1 and pipeline1 from PR2, then PR2 will fail, just because of a bug in PR1. Or even worse, when the situation is reversed and a buggy component can pass due to overwrite.

Solution

Probably the simplest solution here will be to deny concurrent CI/CD jobs. Just process one pull request at a time.

Pull Request 1 -> Pull Request 2 -> ...

With GitHub Actions, it can be done with the concurrency feature. That allows us to define concurrency groups to ensure that only a single job or workflow using the same group will run at a time.
Yes, it can be a bottleneck in the future, especially if the builds take significant time. But it solves the issue and it works for us now. Also, it is quite easy to implement, so let's go further.

Versioning

Issue

Another dangerous area is versions of docker images. We use containerized components that allow us to work with them as with independent applications, it increases reusability, and so on. It means that a ready-to-use component consists of two entities: a specification file and a docker image.
Let's look at what can happen if we always use the image:latest version of the docker image.

Situation 1

For instance, we already have a pipeline specification.

{
  "pipelineSpec": {
    ...
    "deploymentSpec": {
      "component1": {
        "image": "gcr.io/components/component1:latest"
        ...

We do not want to change anything, just want to use it to run the pipeline. Maybe we do it automatically on schedule. At the same time, we can have an ongoing CI/CD job that already rebuilt a part of the components, but some part is still in progress. Outcomes can be unpredictable.

Situation 2

Say two developers work with the same component. Tom updates component1, then works with component2, and at this moment Bet updates component1 as well. When Tom builds pipeline1 the right version of component2 will be used, but the latest image of component1 was overwritten by Bet. Tom will receive something unexpected as result. It is good if the difference between these parallel changes is obvious. Otherwise, it can take a lot of time to understand what is wrong.

Solution

By linking the concrete versions of components with image:version and doing only one CI/CD job at a time, we will be fine in the Staging and Production environments. The situation in the Development environment is still tricky. As an option, we can provide a unique dev environment for each developer. However, it will increase the price and require additional work on setting up and maintaining a variety of environments.
As a solution for the dev, we can add another CLI command (or extend the existing one) to rebuild the whole pipeline with all related components.

In this way, we will avoid the occasional overwriting of components. Everything will be built from the current code and used locally without a chance of interruption.

Conclusion

Here we have considered some important aspects that are not so obvious at the start. However ignoring it can make the CI/CD processes unstable and unpredictable.
Next time we will apply this knowledge to get a new version of CI/CD.

To be continued...

Building CI/CD for Vertex AI pipelines: The first solution

Oleksandr Borodavka — Thu, 07 Jul 2022 11:21:54 +0000

Hey everyone, I am a Senior Data Engineer (MLOps) at FreshBooks. We are currently working on transitioning to fully automated end-to-end ML pipelines. And I want to share some ideas, solutions, and challenges we are dealing with during this journey.

A starting point

We are already building training pipelines with Vertex AI.

Example of Vertex AI pipeline from the GC blog article

That works pretty well. We can include into our pipelines data preparation steps, train several models and choose the best one, dump metadata, send notifications, etc. The Vertex AI pipelines are great themselves, but most of the processes during development and maintaining them are manual.

What's the issue here? Say we already have not just a couple of pipelines but a couple dozen. We probably already have a list of our favourite components which are used in most pipelines and say a bug was fixed in one of them. In this scenario we would need to rebuild all the related pipelines, check them locally, then deploy and check them in a real environment. What if we forgot to rebuild something? Or part of a pipeline was broken after the update? Things become trickier... In addition, all these routine operations take some time.

How can we do better?

Okay, we definitely need some automation here.

Illustration of CI/CD from synopsys

CI/CD practices are widely used in typical software development. And we can use it in our case as well.
To start doing it we need:

Some CI/CD tool, to build and run all the processes. Yeah, we can write some custom scripts for this goal, but there are a lot of stable solutions that we can use.
Automated tests, to check if everything works fine. Spreading changes without quality control is probably not a good idea.

GitHub Actions

GitHub Actions was a good choice for us since we already store all sources in GitHub, making it easy to start working with. There is secret storage and a marketplace with prebuilt actions. You do not need to set up any infrastructure, everything is provided by GitHub for free (with limitations). Alongside that is a mature modern tool that provides many ways of customization, like self-hosted runners and reusable workflows.

Tests

What about tests? Let's firstly answer what we want to test. So, a Vertex AI pipeline is a set of components organized as a DAG. Each component can be presented as a small containerized application. We can test such applications as any other software with Unit tests. Also, we can build a simple isolated pipeline(s) just for one target component to be sure it is configured and built well, and works in the Vertex AI environment. These can be our integration tests for components.
For the pipeline itself, we can add some components inside, to do ML-specific things like model validation. When we run the pipeline in a test environment and all stages of the pipeline are successfully done we can consider it works properly. It will be the integration tests for the pipelines.

Repository structure

There is one more thing we should touch on before writing the CI/CD scripts and that is the organization of code in our repository.



pipelines/
  pipeline1.yaml
  ...
components/
  component1/
    src/
      ...
    tests/
      unit/
          ...
      integration/
          test.yaml
    config.yaml

There is a folder with pipeline specifications and a folder with components. Each component contains source files, tests, and a configuration.
We also built an MLOps Framework to simplify some routines, that is why you see yaml files instead of the usual json. We probably will publish another article about it.

CI/CD workflows

Looks like everything is ready to bring GitHub Actions to the scene.

To build CI/CD in our case we need to be able:

run unit tests for a component
build a component
build a pipeline
run pipelines in different environments

In order to not copy-paste almost the same code, we can use the reusable workflows feature of GHA.

test_component.yml - runs unit tests for a component specified by input parameters



name: Test component

on:
  # the section defines the workflow as reusable and describes the input parameters
  workflow_call:                              
    inputs:
      tests_path:
        required: true
        type: string

jobs:
  tests:
    # run the job on a fresh ubuntu instance
    runs-on: ubuntu-latest
    name: "Run tests for the component"
    steps:
      # get the repo to the instance
      - name: Checkout
        uses: actions/checkout@v3
      # install requirements for the framework
      - name: Install python requirements
        run: make install
      # tun tests using the path to the tests from input parameters
      - name: Run tests
        run: make test ${{ inputs.tests_path }}

build_component.yml - builds a component specified by input parameters



name: Build component

on:
  # the section defines the workflow as reusable and describes the input parameters
  workflow_call:
    inputs:
      component_name:
        required: true
        type: string
    secrets:
      WORKLOAD_IDENTITY_PROVIDER:
        required: true
      SERVICE_ACCOUNT:
        required: true

jobs:
  build:
    # run the job on a fresh ubuntu instance
    runs-on: ubuntu-latest
    name: "Build the component"
    # set permissions necessary for google-github-actions/auth
    permissions:
      contents: "read"
      id-token: "write"
    steps:
      # get the repo to the instance
      - name: Checkout
        uses: actions/checkout@v3

      # Install docker
      - name: Install docker
        run: make install_docker

      # auth to GCP with workload_identity
      # also create credentials_file to use it in the next steps
      - id: "auth"
        name: "Authenticate to Google Cloud"
        uses: "google-github-actions/auth@v0"
        with:
          token_format: "access_token"
          workload_identity_provider: ${{ secrets.WORKLOAD_IDENTITY_PROVIDER }}
          service_account: ${{ secrets.SERVICE_ACCOUNT }}
          create_credentials_file: true
          export_environment_variables: true
      # login to GCR with the token from the previous step
      - name: Login to GCR
        uses: docker/login-action@v2
        with:
          registry: gcr.io
          username: oauth2accesstoken
          password: ${{ steps.auth.outputs.access_token }}

      # install python requirements
      - name: Install requirements
        run: make install
      # run build component command with input parameters
      - name: Build the component
        run: make build component ${{ inputs.component_name }}

Since we build components as docker images we have to install docker on the runner and login to GCR. All necessary credentials are stored in the repository secrets.

run_pipeline.yml - run a pipeline specified by input parameters



name: Run pipeline

on:
  # the section defines the workflow as reusable and describes the input parameters
  workflow_call:
    inputs:
      pipeline_name:
        required: true
        type: string
    secrets:
      WORKLOAD_IDENTITY_PROVIDER:
        required: true
      SERVICE_ACCOUNT:
        required: true

jobs:
  build:
    # run the job on a fresh ubuntu instance
    runs-on: ubuntu-latest
    name: "Run pipeline"
    # set permissions necessary for google-github-actions/auth
    permissions:
      contents: "read"
      id-token: "write"
    steps:
      # get the repo to the instance
      - name: Checkout
        uses: actions/checkout@v3

      # auth to GCP with workload_identity
      # also create credentials_file and export it to environment variables in order to use it in the next steps
      - id: "auth"
        name: "Authenticate to Google Cloud"
        uses: "google-github-actions/auth@v0"
        with:
          token_format: "access_token"
          workload_identity_provider: ${{ secrets.WORKLOAD_IDENTITY_PROVIDER }}
          service_account: ${{ secrets.SERVICE_ACCOUNT }}
          create_credentials_file: true
          export_environment_variables: true

      # install python requirements for framework
      - name: Install python dependecies
        run: make install
      # run "run pipeline" command with input parameters  
      - name: Run pipeline
        run: make run ${{ inputs.pipeline_name }}

build_pipeline.yml looks almost the same so we can skip it and move further.

Great, we have all building blocks that do the main operations. Now we just need to use them in order to build and run pipelines when changes happen. Let's use pipeline1 and component1 from the example above.

build_pipeline1.yml - builds and tests the pipeline1 when it is changed



name: Build pipeline1

on:
  # the workflow can be run manually
  workflow_dispatch:
  # the workflow will be run automatically when a PR is opened to the main branch
  # or changes to the PR related to the pipeline are pushed
  pull_request:
    types: [opened, synchronize, reopened]
    branches:
      - "main"
    paths:
      - "pipelines/pipeline1.yaml"

jobs:
  build:
    # call build_pipeline workflow with target parameters
    uses: ./.github/workflows/build_pipeline.yml
    with:
      pipeline_name: "pipeline1"
    secrets:
      WORKLOAD_IDENTITY_PROVIDER: ${{ secrets.DEV_WORKLOAD_IDENTITY_PROVIDER }}
      SERVICE_ACCOUNT: ${{ secrets.DEV_SERVICE_ACCOUNT }}
  tests:
    # call run_pipeline workflow with target parameters
    # to check it in dev env
    # this step requires the previous step to be done successfully
    needs: build
    uses: ./.github/workflows/run_pipeline.yml
    with:
      pipeline_name: "pipeline1"
    secrets:
      WORKLOAD_IDENTITY_PROVIDER: ${{ secrets.DEV_WORKLOAD_IDENTITY_PROVIDER }}
      SERVICE_ACCOUNT: ${{ secrets.DEV_SERVICE_ACCOUNT }}

build_component1.yml - builds (and tests) the component1 component when it is changed and builds the related pipeline after this



name: Build component1

on:

  # the workflow can be run manually

  workflow_dispatch:

  # the workflow will be run automatically when a PR is opened to the main branch

  # or changes to the PR related to the component1 are pushed

  pull_request:

    types: [opened, synchronize, reopened]

    branches:

      - "main"

    paths:

      - "components/component1/**"

jobs:

  tests:

    # call test_component workflow with target parameters

    uses: ./.github/workflows/test_component.yml

    with:

      tests_path: components/component1/tests

  build:

    # call build_component workflow with target parameters

    # this step requires the previous step to be done successfully

    needs: tests

    uses: ./.github/workflows/build_component.yml

    with:

      component_name: "component1"

    secrets:

      WORKLOAD_IDENTITY_PROVIDER: ${{ secrets.DEV_WORKLOAD_IDENTITY_PROVIDER }}

      SERVICE_ACCOUNT: ${{ secrets.DEV_SERVICE_ACCOUNT }}

  integration-tests:

    # call run_pipeline workflow with target parameters

    # to check the component in dev env

    # this step requires the previous step to be done successfully

    needs: build

    uses: ./.github/workflows/run_pipeline.yml

    with:

      pipeline_name: "component1-test"

    secrets:

      WORKLOAD_IDENTITY_PROVIDER: ${{ secrets.DEV_WORKLOAD_IDENTITY_PROVIDER }}

      SERVICE_ACCOUNT: ${{ secrets.DEV_SERVICE_ACCOUNT }}

  build-pipeline1:

    # call build_pipeline workflow with target parameters

    # this step requires the previous step to be done successfully

    needs: integration-tests

    uses: ./.github/workflows/build_pipeline.yml

    with:

      pipeline_name: "pipeline1"

    secrets:

      WORKLOAD_IDENTITY_PROVIDER: ${{ secrets.DEV_WORKLOAD_IDENTITY_PROVIDER }}

      SERVICE_ACCOUNT: ${{ secrets.DEV_SERVICE_ACCOUNT }}

  test-pipeline1:

    # call run_pipeline workflow with target parameters

    # to check it in dev env

    # this step requires the previous step to be done successfully

    needs: build-pipeline1

    uses: ./.github/workflows/run_pipeline.yml

    with:

      pipeline_name: "pipeline1"

    secrets:

      WORKLOAD_IDENTITY_PROVIDER: ${{ secrets.DEV_WORKLOAD_IDENTITY_PROVIDER }}

      SERVICE_ACCOUNT: ${{ secrets.DEV_SERVICE_ACCOUNT }}

Conclusion

That is it. When changes come the related workflows will be run by GHA. Tests run, components and pipelines rebuild and if everything goes well we will be able to merge the Pull Request and update these changes to Production by running these workflows manually or adding another step to trigger it automatically.

But, are we happy now?