Have you tried multi stage pipelines in Azure DevOps?

Nitharshaan Thevarajah — Sun, 07 Jun 2020 19:01:14 +0000

Do you know how you could make environments look alike with less code? I would like to show you one way that I discovered recently.

I have been able to try out multi stage pipelines in Azure DevOps and I have to say that I'm pretty impressed with some of the possibilities there.
Through this post I'll be highlighting some of the features I think are really good.
I'll assume that you are somewhat familiar with Azure DevOps and pipelines in Azure DevOps.

Features I will be talking about are:

stages
conditions
templates

All of these features together will create a pipeline which is similar to the following multi stage pipeline:

What you need to create a multi stage pipeline in Azure DevOps:

Azure Pipelines
A project with your code which can be uploaded to Azure DevOps
Yaml files for your pipelines

How to structure your yaml file

Without a yaml file you won't be able to get multistage pipelines. Let's look at my sample file which I will use through this post.
Note: I have omitted steps in the Build and Deploy job.

#azure-pipelines.yml

# trigger for this pipeline
trigger:
  - master

# pool used in pipeline
pool:
  vmImage: 'ubuntu-latest'

# global variables
variables:
  artifactName: ReleaseArtifact
  releaseTemplate: 'templates/azure-pipelines.release.yml'

stages:
- stage: CI
  displayName: 'Continuous Integration'
  jobs:
  - job: Build
  ...
  - job: Publish
  ...

- stage: 'Dev'
  displayName: 'Release to Dev'
  dependsOn: CI
  jobs:
  - template: ${{ variables.releaseTemplate }}
    parameters:
      variableGroup: 'Dev'
      environment: 'DEV'
      artifactName: ${{ variables.artifactName }}

- stage: 'Test'
  displayName: 'Release to Test'
  dependsOn: Dev
  jobs:
  - template: ${{ variables.releaseTemplate }}
    parameters:
      variableGroup: 'Test'
      environment: 'TEST'
      artifactName: ${{ variables.artifactName }}

- stage: 'ReleaseA'
  displayName: 'Release to production for Company A'
  dependsOn: Test
  jobs:
  - template: ${{ variables.releaseTemplate }}
    parameters:
      variableGroup: 'Company_A.Release'
      environment: 'COMPANY_A-RELEASE'
      artifactName: ${{ variables.artifactName }}

- stage: 'ReleaseB'
  displayName: 'Release to production for Company B'
  dependsOn: Test
  jobs:
  - template: ${{ variables.releaseTemplate }}
    parameters:
      variableGroup: 'Company_B.Release'
      environment: 'COMPANY_B-RELEASE'
      artifactName: ${{ variables.artifactName }}

This yaml file creates and publishes an artifact in the CI stage. The same artifact is released to Dev, Test and Release environments for company A and company B. Note that the CI stage will always run, while the other stages depends on whether the CI stage succeeds or not.

azure-pipelines.yml works as our main template, while azure-pipelines.release.yml is our release template used to release to each environment.
My environments and variable groups are defined like this:

Stages

A pipeline can be divided into major blocks called stages. By default each stage is run only after the preceding stage is completed.
Each stage is highlighted by a red box in the picture below:

Conditions

Each stage or job can be triggered based on a condition. You can for instance say that a job should run only if previous job is successful and triggered by the master branch:

jobs:
- job: Foo
  steps:
  - script: echo Hello!
    condition: always() # this step will always run, even if the pipeline is canceled

- job: Bar
  dependsOn: Foo
  condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/master')) # this job will only run if Foo succeeds and is triggered by master branch

You also have 'dependsOn' tag where you can specify which jobs or stages the current job or stage depends on.

Templates

Say you would like to do the same release to production for several environments. One way is to duplicate the release stage for each environment. The downside is that you will end up with duplicate code, which we don't want.
This is where templates comes in handy!

In my example yaml file I had this piece of code:

...

variables:
  artifactName: ReleaseArtifact
  releaseTemplate: 'templates/azure-pipelines.release.yml'

...

- stage: 'ReleaseA'
  displayName: 'Release to Production'
  dependsOn: Test
  jobs:
  - template: ${{ variables.releaseTemplate }}
    parameters:
      variableGroup: 'Company_A.Release'
      environment: 'COMPANY_A-RELEASE'
      artifactName: ${{ variables.artifactName }}

- stage: 'ReleaseB'
  displayName: 'Release to production for Company B'
  dependsOn: Test
  jobs:
  - template: ${{ variables.releaseTemplate }}
    parameters:
      variableGroup: 'Company_B.Release'
      environment: 'COMPANY_B-RELEASE'
      artifactName: ${{ variables.artifactName }}

As you can see I'm sending parameters to another file which contains the release jobs. My release template looks like this:

#azure-pipelines.release.yml
parameters:
- name: variableGroup
  type: string

- name: environment
  type: string

- name: artifactName
  type: string

jobs:
  - deployment: DeploySolution
    displayName: 'Deploy solution'
    variables:
      - group: ${{ parameters.variableGroup }} # points to variable group in Azure DevOps
    environment: ${{ parameters.environment }} # points to environments in Azure DevOps
    strategy:
      runOnce:
        deploy:
          steps:
          - task: DownloadPipelineArtifact@2
            displayName: 'Downloading artifact'
            inputs: 
              artifact: ${{ parameters.artifactName }}
              ...
          ...

A template could be seen as a function where you send in variables as paramaters. ${{ }} is in our case used as an expression used to define a variable which is evaluated at compile time or run time.

Summary

Through this post we've been looking at some features introduced with multi stage pipelines in Azure DevOps.
We looked at:

how to structure your yaml file
stages
dependencies
templates and how to send variables as parameters

By using templates, we are forcing each release stage to be equal. This is one way to have environments that do not differ from each other.

These are some of the features I've been able to explore and there are a lot more.